Scanner customization?

[rob-ross]

Regarding scanning the input text, there appear to be two places in the code where some customization ability is suggested for match rules; in the Lexer and JSONPathEnvrionment.

In the Lexer:

    key_pattern = r"[\u0080-\uFFFFa-zA-Z_][\u0080-\uFFFFa-zA-Z0-9_-]*"

    # `not` or !
    logical_not_pattern = r"(?:not\b)|!"

    # && or `and`
    logical_and_pattern = r"&&|(?:and\b)"

    # || or `or`
    logical_or_pattern = r"\|\||(?:or\b)"

and in the JSONPathEnvrionment:

    fake_root_token = "^"
    filter_context_token = "_"
    intersection_token = "&"
    key_token = "#"
    keys_selector_token = "~"
    root_token = "$"
    self_token = "@"
    union_token = "|"

I'm just curious, do you have any use case examples where these values would be different?

E.g., in the Lexer logical_not_pattern allows a match of the bang ! as the standard negation operator in RFC9535, but you have also extended the grammar to allow the use of the word "not". So would a potential customization be changing this to
logical_not_pattern = r"!"
so it now behaves in strict accordance with the RFC? Did you envision other use cases for these? Regarding key_pattern, what kind of customizations can you foresee?

Similarly, for the Env customizations, most of these are part of your RFC extensions. Only root_token and self_token are in the RFC. Does their presence as class variables here imply that they could be re-defined with different symbols? And do you have use cases where your extension characters would need to be customized?

One last thing, I notice that this pattern no longer allows a space in a property name? Is this an extension in your implementation? It's actually more strict than the RFC which allows spaces in name-selectors but not in member-name-shorthand strings. I seem to recall there used to be a space here, because I had commented on your code allowing spaces in member-name-shorthand strings. I wasn't criticizing this, I was just noting it. But now your name-selectors also won't allow spaces, whereas they are allowed in the RFC. Personally I'm fine with this extension. That would allow for this JSON value:

{
   "foo bar" : true
}

Both $.foo bar and $["foo bar"] to resolve to the value "true".

[jg-rp]

I'm just curious, do you have any use case examples where these values would be different?

Yes. I have JSONPath integrated with a template engine. That template engine uses and, or and not, and disallows &&, || and ! . It is also more restrictive when it comes to property/key/variable names.

So would a potential customization be changing this to logical_not_pattern = r"!"

Yes. Changes like that would bring python-jsonpath closer to RFC 9535. The defaults are kept as they are so we don't break dependent projects that started using python-jsonpath before RFC 9535 compatibility was on the radar.

Does their presence as class variables here imply that they could be re-defined with different symbols? And do you have use cases where your extension characters would need to be customized?

They could be redefined, but I've not seen it done in the wild. I think you can be quite selective as to which symbols can be overridden in your port.

I notice that this pattern no longer allows a space in a property name? Is this an extension in your implementation?

Quoted property names are scanned as TOKEN_DOUBLE_QUOTE_STRING or TOKEN_SINGLE_QUOTE_STRING and are validated in the parser with RE_INVALID_NAME_SELECTOR, which does allow whitespace.

from jsonpath import findall

data = {"foo bar": True}

print(findall("$['foo bar']", data))  # [True]
print(findall('$["foo bar"]', data))  # [True]
print(findall("$.foo bar", data))  # []

That last one is being parsed as $["foo"]["bar"] and should be a syntax error.

[rob-ross]

re: "$.foo bar"

I just meant that if you included a space in the "key_pattern" regex, then you could write $.foo bar (with a space) and that would match the key "foo bar". Of course, I have no idea if this would break other areas of the grammar. Although off the top of my head, a dot followed by a string then a space then another string doesn't have any other valid grammar rule that I can remember, so it seems like it would unambiguous and able to be parsed as a key name that contains space.

Again - I'm not saying this is something that should be allowed, just pointing out this possibility. On second thought, this might be inconsistent with other libraries like JSON Pointer/Patch, so there could be other reasons not to support this syntax.

[jg-rp]

Ah, yes. I understand now. 👍