Skip to content

Conversation

dandavison
Copy link
Contributor

@dandavison dandavison commented Sep 12, 2024

Closes #1633

As in temporalio/sdk-python#617 and temporalio/cli#650, I propose to modify the test suite to output junit XML. The motivation for this is that, given appropriate tools, it will make it easier for us to study patterns of flakiness, test timings etc, across matrix builds within a run, and across runs. Producing the files now will allow us to test tools for doing this.

@dandavison dandavison force-pushed the junit-xml branch 3 times, most recently from 4152931 to f83ff64 Compare September 12, 2024 10:17
@dandavison dandavison marked this pull request as ready for review September 12, 2024 13:49
@dandavison dandavison requested a review from a team as a code owner September 12, 2024 13:49
@dandavison dandavison marked this pull request as draft September 12, 2024 14:02
@@ -1,20 +1,29 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c h1:pxW6RcqyfI9/kWtOwnv/G+AzdKuy2ZrqINhenH4HyNs=
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All changes to go.mod and go.sum are the result of running go get gotest.tools/gotestsum (looks like it was already a transitive dependency IIUC).

@dandavison dandavison marked this pull request as ready for review September 12, 2024 14:51
@dandavison
Copy link
Contributor Author

@Quinn-With-Two-Ns To output junit XML I had to switch to using gotestsum. One question here is are you OK with the new test output format from gotestsum? From a quick look it appears that one big difference is that output from successful tests is hidden by default. I was optimistic that we'd be OK with this switch since https://github.com/temporalio/temporal uses it, and it looks like it's quite configurable, and fairly widely used by high profile Go projects: https://github.com/gotestyourself/gotestsum

@Quinn-With-Two-Ns
Copy link
Contributor

No concern about the test format. Question about the general PR though, from the issue it says

The motivation for this is that, given appropriate tools, it will make it easier for us to study patterns of flakiness

Does that appropriate tool exist yet today?

@dandavison
Copy link
Contributor Author

Does that appropriate tool exist yet today?

Good question. I couldn't find something that would work well, e.g. correlating tests across our matrix builds, so I've started creating our own from scratch. I'm not quite ready to have others test it (but absolutely welcome any help developing it).

TEMP: just install gotestsum in ci.yml

Invoke gotestsum correctly

go get gotest.tools/gotestsum

Install gotestsum locally

Pass file stem from ci.yml; allow build script to control output dir

Dump junit XML for cloud test also

Delete unnecessary quotes

Add comment on duplicated hardcoded path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Modify test suite to dump junit XML
3 participants