fix: discrepancy on leading 0's with cpp nix's behavior.

[hsjobeki]

Summary & Motivation

As described in #157, Nix doesn't allow additional leading 0's on floats. Adding additional leading zeros to numbers may even break them into two tokens. (TOKEN_INT and TOKEN_FLOAT) I've also described further information in the referenced issue.

This means if we closely follow c++ Nix's behavior, with this PR, rnix tokenizes as follows. (Which represents Nix's behavior)

01.1e2
->

TOKEN_INTEGER, "01"
TOKEN_FLOAT, ".1e2"

01.e2
->

TOKEN_INTEGER, "01"
TOKEN_DOT, "."
TOKEN_IDENT, "e2"

This may fail to parse/evaluate, but we don't care at the tokenizer level.

nix-repl also shows that Nix tokenizes the same way.

nix-repl> 01.e2
error: value is an integer while a set was expected

@tazjin The error here may need improvement. But we can handle that further, e.g., in the tvix compiler. To give better error messages than C++ nix.

Backwards-incompatible changes

All tests on floats were run successfully. Since this only adds custom handling on leading 0's, this doesn't break any backward compatibility.

Further context

Solves #157

[hsjobeki]

This PR can be closed.
As soon as NixOS/nix#8605 is fixed, we don't have any discrepancies. rnix tokenized correctly already.