Skip to content
This repository was archived by the owner on Jan 17, 2024. It is now read-only.

Conversation

@ngie-eign
Copy link
Contributor

@ngie-eign ngie-eign commented Aug 20, 2023

This change adds fine grained execution of GoogleTest framework-based test
programs.

First, the GoogleTest test is executed with --gtest_list_tests. Next,
based on the output from --gtest_list_tests, the testcases are run
individually. Finally, the output from each unique testcase is scraped
and digested by Kyua in a format that Kyua supports.

The output from each unique testcase is based on the standard output it
provides, per the test output protocol defined in the GoogleTest docs on
github [1], [2], and instrumented via various demo programs I created,
which can be found on GitHub in my scratch repo.

This is initial framework integration. There're additional improvements that
can be made by leveraging either the JSON or XML structured output format,
instead of scraping standard output using beginning and ending sentinels to
search for regular expressions. In order to do that though without reinventing
the wheel, Kyua would need to rely on an external JSON or XML library.

This test interface supports fine grained execution of test programs
like the ATF test interface, but matches behavior of plain/TAP interfaces by
instead supporting metadata passing via $TEST_ENV_ prefixed environment
variables.

This support adds additional tests for verifying pass, fail, skip (available in
GoogleTest 1.9.0+), and disabled result determination, using mock output and a
mock test program (engine/googletest_helpers), for parity with other test
formats (ATF, plain, TAP). The helper test program purposely avoids relying on
getopt_long* for portability reasons, and the GoogleTest test infrastructure,
in order to limit Kyua's dependencies.

As part of this change, store/read_transaction.cpp needed to support
optional reasons provided with skip results. While it's bad form to omit
test results with tests, providing a reason is optional with GoogleTest, and
unfortunately not all portions of the test framework output a reason when
GTEST_SKIP() is called. See the issue in [3] for one such example issue when
GTEST_SKIP() is called from SetUp test fixtures.

  1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md
  2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md
  3. GTEST_SKIP() << {message} doesn't output message when executed from SetUp() fixtures google/googletest#2208

Signed-off-by: Enji Cooper [email protected]

@ngie-eign ngie-eign force-pushed the googletest-support__freebsd branch from e4a04d5 to f8a1bdd Compare August 20, 2023 22:47
This change adds fine grained execution of GoogleTest framework-based test
programs.

First, the GoogleTest test is executed with `--gtest_list_tests`. Next,
based on the output from `--gtest_list_tests`, the testcases are run
individually. Finally, the output from each unique testcase is scraped
and digested by Kyua in a format that Kyua supports.

The output from each unique testcase is based on the standard output it
provides, per the test output protocol defined in the GoogleTest docs on
github [1], [2], and instrumented via various demo programs I created,
which can be found on GitHub in my [scratch repo](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest).

This is initial framework integration. There're additional improvements that
can be made by leveraging either the JSON or XML structured output format,
instead of scraping standard output using beginning and ending sentinels to
search for regular expressions. In order to do that though without reinventing
the wheel, Kyua would need to rely on an external JSON or XML library.

This test interface supports fine grained execution of test programs
like the ATF test interface, but matches behavior of plain/TAP interfaces by
instead supporting metadata passing via `$TEST_ENV_` prefixed environment
variables.

This support adds additional tests for verifying pass, fail, skip (available in
GoogleTest 1.9.0+), and disabled result determination, using mock output and a
mock test program (`engine/googletest_helpers`), for parity with other test
formats (ATF, plain, TAP). The helper test program purposely avoids relying on
`getopt_long*` for portability reasons, and the GoogleTest test infrastructure,
in order to limit Kyua's dependencies.

As part of this change, `store/read_transaction.cpp` needed to support
optional reasons provided with skip results. While it's bad form to omit
test results with tests, providing a reason is optional with GoogleTest, and
unfortunately not all portions of the test framework output a reason when
`GTEST_SKIP()` is called. See the issue in [3] for one such example issue when
`GTEST_SKIP()` is called from SetUp test fixtures.

1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md
2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md
3. google/googletest#2208

Signed-off-by: Enji Cooper <[email protected]>
@lwhsu
Copy link
Member

lwhsu commented Jan 15, 2024

The canonical kyua repo has been moved to freebsd/kyua, I'd like to close the PRs in this repo and archive it soon. @ngie-eign do you mind to move this PR to freebsd/kyua and continue the discussion there?

@ngie-eign ngie-eign closed this Jan 16, 2024
@lwhsu
Copy link
Member

lwhsu commented Jan 17, 2024

ref: freebsd/kyua#203

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants