feat(api): ensure StackRunConfig

[cdoern]

What does this PR do?

StackRunConfig is part of our public API, ensure stability of this datatype using a pytest snapshot test.

If the pydantic model changes, it will fail. A snapshot can be re-generated via @github-actions regenerate snapshots by a code owner.

The API conformance test will then re-run and pass.

added json_schema_extra to two spots in the run config so user paths do not get propagated into the snapshot json

Test Plan

API Conformance Test should pass.

[cdoern]

working on this -- might convert to live in tests/

[cdoern]

uh oh, pre commit seems broken still?

[leseb]

Overall, I like the approach. I have a few questions:

• how do we intend to use this? Do we bump the version each time a new change to the schema is introduced?
• How do we refresh the tests/api/snapshots/test_models/test_run_config_v1_schema_is_unchanged/stored_run_config_v1_schema.json file? Should pre-commit do it? Should we run the scripts/api-conformance.sh script with a special flag to rehydrate?

I think we need some guidelines, not just a script that fails the CI :) (unless I missed something!)

Thanks!

[leseb]

can we use a more accurate name for the script?

[cdoern]

@leseb , I made it so that when a breaking change is acknowledged through the proper process ! or BREAKING CHANGE:, the pydantic test is run and always regenerates the schema and does not fail.

Also changed the name of the script to test_pydantic_models

[cdoern]

FYI I need to rebase this each time the run config changes, sorry for the churn :)

[leseb]

can you remove tests/api/snapshots/test_models/test_run_config_v1_schema_is_unchanged/stored_run_config_v1_schema.json?

[leseb]

why don't we run this in pre-commit?

[cdoern]

I think keeping it in the conformance yml makes sense though so it can get the context of the ! or BREAKING CHANGE: from this workflow. In pre-commit people will just override the snapshots locally and repush.

[leseb]

same command regardlesss if REGENERATE_SNAPSHOT is set?

[cdoern]

made this:

# Run pytest with snapshot testing
echo "=== Running Snapshot Tests ==="
  if [[ "$REGENERATE_SNAPSHOT" == "true" ]]; then
    echo "Regenerating snapshots..."
    pytest -s -v "$SCRIPT" --snapshot-update
    exit 0
  else
    pytest -s -v "$SCRIPT"  # do not update snapshots.
  fi

so that the test only updates the snapshots and passes if the proper ack is given. The exit 0 is necessary because even when updating snapshots these types of tests fail on the diff I think.

the other way will fail if the snapshots differ.

[cdoern]

this is a re-written version of precommit-trigger. I used git mv but because the diff is so large it's registering it as a new file annoyingly. Sorry

[cdoern]

moving to draft while I wrestle with the new action I am introducing.