-
Notifications
You must be signed in to change notification settings - Fork 48
Adding assert_equals_golden
and assert_output_equals_golden
#67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Adding support for "golden files". Golden files are (often text) files that are checked into a repo and are used in testing for asserting strings against their content. They have a number of useful properties that make them a great alternative to single (possibly repetitive) assertions in files. Some benefits include: * WYSIWYG plaintext output assertions (separating asserted output from test case logic). * Test failure output that is searchable (asserted output is checked into repo as files). * Clear file diffs of test assertions during merge / code review. * Terse assertions in test cases (single assert instead of many verbose `assert_line` and `refute_line` for every line of output). * Reusable golden files (declared once, used for many test cases). * Clear intention (same exact expected output) when asserted against same goldens in multiple test cases. * Can be clearly diff'd across multiple lines in failure message(s). * Easily updated. This PR adds support to bats for golden files with the `assert_equals_golden` and `assert_output_equals_golden` test helper functions. `assert_equals_golden` takes 2 arguments, <actual> and <golden file path>, to assert <actual> is the same as the contents in <golden file path>. `assert_output_equals_golden` takes 1 argument, <golden file path>, to assert `output` is the same as the contents in <golden file path>. These functions support a number of features, including `--regexp` and automatic updating of golden files (via setting the `BATS_ASSERT_UPDATE_GOLDENS_ON_FAILURE` environment variable). Golden file assertions will properly account for empty lines or trailing newlines (e.g. when asserting against output obtained by `run --keep-empty-lines`). This PR adds `assert_equals_golden.bats` which contains test cases for the newly added functionality. Note that the output of these functions (that is asserted on in tests) has incorrect "lines" count. This is due to an incompatibility between `run --keep-empty-lines` and how various bats-support helper functions work. See bats-core/bats-support#11.
…ailed (using `$output` instead of passed string). * Updating `assert_equals_golden` tests in `test/assert_equals_golden.bats` to invalidate `output` and use another variable instead (to better catch any similar issues).
* Adding new test cases for `assert_file_equals_golden`.
* Adding `\` to list of special characters in doc strings. * Adding test cases to cover all special characters with `BATS_ASSERT_UPDATE_GOLDENS_ON_FAILURE`. * Cleaning up some unnecessary lines in a few existing `assert_equals_golden` tests.
…even if the golden ends up being updated). * Adding prefix of golden file path before error messages. * Simplifying return status from assert functions. * Updating tests to assert output with new changes. * Adding helper function to save tmp file path used in test cases (to then be used in assertion messages). * Adding a space after `\` in some test cases to handle/workaround `\\\n` in docstrings not being parsed properly by bash.
Is there a strong case for having a dedicated matcher instead of using I recognize some other conveniences still need built out (better diffing with the But I would propose we beef up the core assertions and then use them for what is still a text matching assertion. If we do want some additional helper, I could see a fixture helper of some kind that had conventional pathing built-in. But the assertion side of the equations doesn't feel dissimilar enough from the core methods to necessitate a new one. (Some evidence to support this is that many other test utilities in other frameworks have plenty of helpers for fixtures in general; but none that I've seen ever combine fixture support with the matcher.) |
Using I made these into separate assertions to both allow for clarity of how the assertion is done and to keep existing functions from being bloated. When I had added in I've seen libraries with golden file support and sometimes automatic updates (which must be based on the text asserted) is done via built-in functionality or via scripts. But it's usually a common desire when working with them. |
… set to 1 if set at all.
…f` (to work appropriately with `set -e`).
…, compatible args, file paths, etc) into helper functions.
* Message no longer prints contents (as it will be printed after). * No longer performs early return. * Still prints full assertion message and optionally does golden update (when enabled).
…n_file_contents`. * Consolidating all assertion error messages into helper functions. * Consolidating all updated golden messages into helper function. * Creating `_assert_golden_read_file_contents` general function. * `_assert_golden_read_golden_file_contents` now uses `_assert_golden_read_file_contents` (consolidated logic). * Updating `assert_file_equals_golden` to use `_assert_golden_read_file_contents`.
* Fixing more fail-able commands by putting directly in `if`.
* Simplifies output streams. * Properly handles trailing newlines.
…te_golden_file_contents", so then it is only called when they succeed.
* Fixing file path validation check in `assert_file_equals_golden`. * Updating invalid regex error message to contain golden file name and contents.
* Fixing assertion for argument count test cases on `assert_file_equals_golden`.
Adding support for "golden files".
Golden files are (often text) files that are checked into a repo and are used in testing for asserting strings against their content. They have a number of useful properties that make them a great alternative to single (possibly repetitive) assertions in files.
Some benefits include:
assert_line
andrefute_line
for every line of output).This PR adds support to bats for golden files with the
assert_equals_golden
,assert_output_equals_golden
, andassert_file_equals_golden
test helper functions.assert_equals_golden
takes 2 arguments,<actual>
and<golden file path>
, to assert<actual>
is the same as the contents in<golden file path>
.assert_output_equals_golden
takes 1 argument,<golden file path>
, to assertoutput
is the same as the contents in<golden file path>
.run command_under_test assert_output_equals_golden 'test_case_golden_file.txt'
assert_file_equals_golden
takes 2 arguments,<target file path>
and<golden file path>
, to assert that the contents of the given file is the same as the contents in<golden file path>
.These functions support a number of features, including regexp and automatic updating of golden files (via setting the
BATS_ASSERT_UPDATE_GOLDENS_ON_FAILURE
environment variable).Golden file assertions will properly account for empty lines or trailing newlines (e.g. when asserting against output obtained by
run --keep-empty-lines
).The alternative is
$(cat test_case_golden_file.txt)
with a lot of extra boilerplate, edge case handling, and manually updating golden files.This PR adds
assert_equals_golden.bats
which contains test cases for the newly added functionality.Extra Note: the output of these functions (that is asserted on in tests) has incorrect "lines" count. This is due to an incompatibility between
run --keep-empty-lines
and how various bats-support helper functions work. See bats-core/bats-support#11 for more detail on this caveat.