fix scope related bug

[eddycharly]

This test should fail:

	{
		args: args{
			expression: `let({"b": @.a}, &a.b)`,
			data: map[string]interface{}{
				"a": 42.0,
			},
		},
		want: 42.0,
	}

Once &a has been looked up from the scope, we should not use the scope again to lookup .b.

@springcomp am I right thinking we have a bug here ?

[springcomp]

@eddycharly that’s a good question and my interpretation of the spec (which I did not write) is that it should still be using the scope up for the entire duration of the let() evaluation.

So indeed:

• a -> 42.0
• b
  • Is evaluated based on the current context, which returns null
  • Therefore, it is looked up in the scope, and is found to be 42.0

So I do not think that is a bug.
But there is definitely room for interpretation, which is not satisfying.

[springcomp]

The non specified behaviour is actually triggered by this part of the spec:

Parent scopes are created by nested let() calls.

Which implicitely can be interpreted as if a scope has a lifetime tighed to the entire duration of the let() function invocation.

[eddycharly]

My interpretation is that once the scope chain was used to lookup something, this something becomes the only thing available for further processing.

With let({"b": @.a}, &a.b):

• a -> no lookup has occurred before (current context is { a: 42 } and scope chain is { b: { a: 42 } }), we lookup a, current context becomes 42 and scope chain becomes null
• .b -> there's no .b in 42, result is null

From what I understand scope chain applies only until something is looked up from the chain.

How can we clarify this point of the spec ?

[eddycharly]

I'm definitely not comfortable with the current behaviour :(

[springcomp]

From what I understand scope chain applies only until something is looked up from the chain.

For clarity, my understanding of your interpretation is that the scope chain should apply:

• Only until something is looked up from the chain, and returns successful evaluation,
• At the end of the current invocation for let() one level is discarded from the stack and the scope is available again.

Consider this compliance tests (which I did not write either):

Given:

{
  "nested": {
    "a": {
      "b": {
        "c": {
          "fourth": "fourth"
        },
        "third": "third"
      },
      "second": "second"
    },
    "first": "first"
  }
}

The following expression:

nested.let({level1: first}, &a.let({level2: second}, &b.let({level3: third}, &c.{first: level1, second: level2, third: level3, fourth: fourth})))

Returns:

{
  "first": "first",
  "second": "second",
  "third": "third",
  "fourth": "fourth"
}

In the third nested level, the multi-hash expression tries to evaluate the expression:

{ first: level1, second: level2, third: level3, fourth: level4 }.

• level1 is not found in the current context, therefore is looked up in the scope chain.
• level1 is found by walking up the scope chain and evaluates to first.

Is your interpretation so that at this point, the scope chain should be considered as not available for the rest of the evaluation?
Or is your interpretation more in line with the following nested level sub-expression:

{ first: level1.level2, second: level2, third: level3, fourth: level4 }

Returning:

{
  "first": null,
  "second": "second",
  "third": "third",
  "fourth": "fourth"
}

I think that’s a point worth clarifying indeed.

How can we clarify this point of the spec ?

We need to discuss the JEP with the wider community – although there is not much traction at the moment.
The tricky thing is that the spec has been designed by James who appear to no longer provide feedback.

[eddycharly]

Kindly ping, @jamesls would you mind clarifying the expected behaviour here ? 🙏 🤞

[springcomp]

@mtdowling 🙏

[springcomp]

@eddycharly for context:

jmespath/jmespath.site#6

This bit in particular addresses your concerns.

[eddycharly]

@springcomp thanks for the links !

Unfortunately I still find the current behaviour quite confusing.

The way we stack scopes is pretty clear but the way we use the scopes to resolve fields is not.
I see proposals in the discussion to use a dedicated token but it seems to bring other problems too.

I wonder if it would make sense to use a function to enable lookup in scopes instead 🤔

[eddycharly]

In my understanding, the result of evaluating the expression:

nested.let({level1: first}, &a.let({level2: second}, &b.let({level3: third}, &c.{first: level1, second: level2, third: level3, fourth: fourth})))

would be:

{
  "first": null,
  "second": null,
  "third": null,
  "fourth": "fourth"
}

because &c.{...} is looking up c which is found in the current context and overrides scopes previously defined.

[eddycharly]

This could be rewritten to:
nested.let({level1: first}, &a.let({level2: second}, &b.let({level3: third}, &c.let({level4: fourth}, &{first: level1, second: level2, third: level3, fourth: level4}))))

To get the expected behaviour though.

[springcomp]

While I agree the current behaviour with sub-expression is unclear, I think it’s reasonable and intuitive to expect a multi-select-hash to be thought of as evaluating disctint expressions, each one having access to the original scope and context. A bit like evaluating each argument to a function does not make the context suddenly disappear / be more restricted for subsequent function arguments.

Specifically, I would expect the full original context and scope chain to be available when evaluating:

• Each property of a multi-select-hash
• Each element of a multi-select-list
• Each argument of a function
• Each evaluating stream in a projection
• etc.

[eddycharly]

We could also use a function for that:
nested.let({level1: first}, &a.let({level2: second}, &b.let({level3: third}, &c.ref(&{first: level1, second: level2, third: level3, fourth: fourth}))))

In the expression above the ref function is used to enable scope lookups.