Delta variant of testchanges

[joswarmer]

I started with creating some Delta versions of the testchanges, can you take a look and see whether this is a good way forward?

[dslmeinte]

I started with creating some Delta versions of the testchanges, can you take a look and see whether this is a good way forward?

Shouldn't these cases go somewhere under the delta/ directory? That way, these files should get validated against the JSON Schema as well.

[joswarmer]

I started with creating some Delta versions of the testchanges, can you take a look and see whether this is a good way forward?

Shouldn't these cases go somewhere under the delta/ directory? That way, these files should get validated against the JSON Schema as well.

I put them in the testchanges directories, because the delta's are exactly the same testcase/data as the bulk ones in there. They have the same result as the bulk testcases in the same directory.

[enikao]

I think this is the same as on branch niko/delta-api, although my branch also contains C# stuff. I'm about to separate that.

[enikao]

Actually, I think they are already merged: https://github.com/LionWeb-io/lionweb-integration-testing/tree/main/delta

[enikao]

I'm cleaning up my branching mess, so you probably need to rebase on main.

[enikao]

I'm wondering how useful such simple, one-message tests are (besides having an example of each message).

According to the protocol, we always have to sign on first, and send our participation id -- which is handed out by the repository.
Also, delta only gets interesting once we have several clients, so we had to mark somehow which message is sent by which client -- and in which order, because that's also very important. We probably also want to assert the current state of the partition somewhere in between.

Putting all of this in pure text files results in a pretty complex format. I tend to just write such tests as code, and then pit the different client/repo implementations against each other. The C# tests are prepared for that: They always start their counterpart as external process via a command-line call, so it should be pretty easy to e.g. use the C# client tests to run against the lw-server repo.

[dslmeinte]

I'm wondering how useful such simple, one-message tests are (besides having an example of each message).

According to the protocol, we always have to sign on first, and send our participation id -- which is handed out by the repository. Also, delta only gets interesting once we have several clients, so we had to mark somehow which message is sent by which client -- and in which order, because that's also very important. We probably also want to assert the current state of the partition somewhere in between.

Putting all of this in pure text files results in a pretty complex format. I tend to just write such tests as code, and then pit the different client/repo implementations against each other. The C# tests are prepared for that: They always start their counterpart as external process via a command-line call, so it should be pretty easy to e.g. use the C# client tests to run against the lw-server repo.

I agree with the general usefulness – or lack thereof – of tests based on single messages and no “in scenario” assertions.

However, it's a relatively quick win to validate these messages at least against the prevailing JSON Schema and preferably against as much of the specification as possible. We've seen – and are seeing – relatively simple (to fix) non-conformance to the spec often enough.

[enikao]

However, it's a relatively quick win to validate these messages at least against the prevailing JSON Schema and preferably against as much of the specification as possible. We've seen – and are seeing – relatively simple (to fix) non-conformance to the spec often enough.

We're already doing this, I think:

lionweb-integration-testing/.github/workflows/build.yaml

Lines 12 to 16 in e6c7648

	- name: Validate delta JSON
	uses: cardinalby/schema-validator-action@v3
	with:
	schema: 'delta.schema.json'
	file: 'delta/**/*.json'

[dslmeinte]

However, it's a relatively quick win to validate these messages at least against the prevailing JSON Schema and preferably against as much of the specification as possible. We've seen – and are seeing – relatively simple (to fix) non-conformance to the spec often enough.

We're already doing this, I think:

lionweb-integration-testing/.github/workflows/build.yaml

Lines 12 to 16 in e6c7648

	- name: Validate delta JSON
	uses: cardinalby/schema-validator-action@v3
	with:
	schema: 'delta.schema.json'
	file: 'delta/**/*.json'

For all files under delta/, yes, but @joswarmer added files under testchanges/data/ — hence the question whether these files should be under delta/ or explicitly validated in their current location.

[joswarmer]

I'm wondering how useful such simple, one-message tests are (besides having an example of each message).

Of course these tests will not be enough, but I want to use them as unit tests to test each delta command in isolation. This is the first level of unit tests that is needed. For these tests all the startup messages can be inserted by the test code.
This way we know that the server implements individual delta's correctly.

After that we can add tests for more complex scenarios.

[enikao]

I'm wondering how useful such simple, one-message tests are (besides having an example of each message).

Of course these tests will not be enough, but I want to use them as unit tests to test each delta command in isolation. This is the first level of unit tests that is needed. For these tests all the startup messages can be inserted by the test code. This way we know that the server implements individual delta's correctly.

After that we can add tests for more complex scenarios.

If the issue is the actual content of the examples (so they fit with your tests), feel free to edit the existing ones -- their content is arbitrary.