Weird regexps

[paolopas]

Recently there has been interest in line 399 of the src/tools/python.jam file, to make sure things were working as they should I took a look and got a nasty surprise, in the following Jamfile I report the test that runs the python.jam along with other instructions that demonstrate that the regex engine does not behave exactly as one would expect.

local version = "12345." ;
echo A [ MATCH "([0-9]+)\.([0-9]+)(.*)" : $(version) ] ;
echo B [ MATCH "([0-9]+\.[0-9]+)(.*)" : $(version) ] ;
echo C [ MATCH "([0-9]+)\.([0-9]*)(.*)" : $(version) ] ;
echo D [ MATCH "([0-9]+\.[0-9]*)(.*)" : $(version) ] ;

local major-minor = [ MATCH "^([0-9]+)\.([0-9]+)(.*)$" : $(version) ] ;
if ! $(major-minor[2]) || $(major-minor[3])
{
	ECHO "Warning: expects a two part (major, minor) version number; got" $(version) instead ;
}

local version = "67890" ;
echo E [ MATCH "([0-9]+)\.([0-9]+)(.*)" : $(version) ] ;
echo F [ MATCH "([0-9]+\.[0-9]+)(.*)" : $(version) ] ;
echo G [ MATCH "([0-9]+)\.([0-9]*)(.*)" : $(version) ] ;
echo H [ MATCH "([0-9]+\.[0-9]*)(.*)" : $(version) ] ;

local major-minor = [ MATCH "^([0-9]+)\.([0-9]+)(.*)$" : $(version) ] ;
if ! $(major-minor[2]) || $(major-minor[3])
{
	ECHO "Warning: expects a two part (major, minor) version number; got" $(version) instead ;
}

A , B, E, F, G, H cases should not give a match

# b2
A 123 5 .
B 12345 .
C 12345  
D 12345. 
Warning: expects a two part (major, minor) version number; got 12345. instead
E 678 0 
F 67890 
G 6789  
H 67890 
...found 1 target...

regexeps deserve a few more tests

[paolopas]

I took a quick look and realized that the problem is not related to the regexps.

First of all, it must be said that the exact same behavior occurs if you write a test in Python, but this is not a given since lately I've noticed that the executable and the tests are starting to diverge. But that's another problem, perhaps even more difficult than this one.

The point here is that some of the 200 billion strings crammed into the code decided to eat the escape characters (\). As could also be deduced by looking at the test results.

The proof is that if you replace the \. (which should match the dot) in the regexps with [.] or \\\. then you get the expected results, that is

# b2
A
B
C 12345  
D 12345. 
Warning: expects a two part (major, minor) version number; got 12345. instead
E
F
G
H
Warning: expects a two part (major, minor) version number; got 67890 instead
...found 1 target...

[paolopas]

Here's my final verdict:

1. I'm getting older.
2. the regexp in python.jam is wrong! And unfortunately it's in good company; it'll take a while to fix them all...
3. B2 regexps are innocent, but they too need some work. Which ?

• There are a few too many NULLs floating around in the code and other things that are a bit too "C", not properly type-safe
• Error handling needs to be improved, not only in the messages issued, for example the following Jamfile

MATCH \\"f...o\\" : furbo ;

results in

# b2
re error trailing \
Segmentation fault (core dumped)

Which is nothing like what the scanner does for a similar error:

MATCH \\"f...o\" : furbo ;

# b2
Jamroot:1: unmatched " in string at argument MATCH
Jamroot:1: syntax error at EOF
...found 1 target...

• The usability of the regexp class also needs improvement.

4. We need to add some tests for regexps.

In short, several PRs are coming soon.