Skip to content

Reduce test flakes #8989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 52 commits into
base: main
Choose a base branch
from
Open

Reduce test flakes #8989

wants to merge 52 commits into from

Conversation

leftwo
Copy link
Contributor

@leftwo leftwo commented Sep 4, 2025 •
edited
Loading

Various tuning to reduce test flakes during CI.

Decrease the number of tests running at the same time on buildomat runs.
The buildomat AWS instance has 8 cores, so we configure the nextest run to have
two less than that. This reduces overall system load.

Increase some internal timeouts, and add loops to some tests to allow them to finish
even if the system is under heavy load.

Added a few debug messages, but, they may be of limited value.

leftwo added 30 commits August 29, 2025 20:00
@leftwo
Add debug to test flake step
f2c6334
@leftwo
more info when failing
889ab2d
@leftwo
more debug logging
93c2bec
@leftwo
more debug
d251d45
@leftwo
more debug on more tests
b1b197b
@leftwo
More debug
6f129c4
@leftwo
cargo fmt
b002fb2
@leftwo
make some checks loop forever, others take longer
b7ba833
@leftwo
loop around checking clean slate/ replacement state
f4157a5
@leftwo
fixed incorrect check on last commit
459fcaa
@leftwo
Make loop forever for volume delete check
b82f35d
@leftwo
fix variable name
1be06ae
@leftwo
Make another test loop forever
5fa251a
@leftwo
more debug, correct pre-test checks
04051f3
@leftwo
print more when test_cooldown_on_sub_reinc fails
5c28a2c
@leftwo
don't unwrap when debugging possible failures
fadd5a0
@leftwo
debug for test_quiesce_full
967a0d4
@leftwo
Increase wait time for instance changes to 320
deb35db
@leftwo
more debug, but this ones a problem
0478bbd
@leftwo
Merge branch 'main' into alan/why-do-tests-flake
3ae245c
@leftwo
Merge branch 'main' into alan/why-do-tests-flake
43dd58f
@leftwo
Merge branch 'main' into alan/why-do-tests-flake
cd16660
@leftwo
Give up eventually on assert_region_snapshot_replacement_request_unto…
146023c 
…uched
@leftwo
Add a comment about a possible flake point
f5e7e7a
@leftwo
Increase timeout for crucible replacements
0cff9d8
@leftwo
reduce test threads
1d2b8ba
@leftwo
Bump timer for inventory collection
ad9163c
@leftwo
Increase timeout for pantry_endpoints to 120 seconds
a9f7450
@leftwo
Cleanup test flake comments
e9e8c03
@leftwo
Remove debugging statements
46e621d
@leftwo leftwo mentioned this pull request Sep 6, 2025
Open
leftwo and others added 19 commits September 6, 2025 06:00
@leftwo
comment cleanup
ff71337
@leftwo
More comment changes
9d5d02e
@leftwo
Comment for quiesce eprintln
e45b4e5
@leftwo
Remove more debug messages
5e83e94
@Nieuwejaar @leftwo
Update dendrite, lldpd, maghemite, and sidecar-lite to get ipv6 ecmp (#…
35aa542 
…8987)
@KaiSforza @leftwo
nix: fix flake versions (#8969)
d13dfcf 
Rust 1.89 wasn't available on that nixpkgs revision, but it was
specified in the `rust-toolchain.toml` file.
@JustinAzoff @leftwo
Explain how to authenticate to the now running control plane. (#8977)
67beca5
@david-crespo @leftwo
Bump web console (fix IP pools) (#9007)
6fd635c 
oxidecomputer/console@8cc5b25...e0efddf

* [e0efddfe](oxidecomputer/console@e0efddfe)
oxidecomputer/console#2897
* [33c4c433](oxidecomputer/console@33c4c433)
oxidecomputer/console#2895
* [2e4109a1](oxidecomputer/console@2e4109a1)
oxidecomputer/console#2893
@rcgoodfellow @leftwo
bump maghemite to pick up #539 (#9009)
2151b30
@leftwo
remove debug messages
044713a
@Nieuwejaar @leftwo
Update dendrite, lldpd, maghemite, and sidecar-lite to get ipv6 ecmp (#…
3c4736c 
…8987)
@leftwo
Merge branch 'main' into alan/why-do-tests-flake
666d5dd
@leftwo
Remove more debug logs
fca7c7b
@leftwo
Remove another debug print, run tests again
732cbb1
@leftwo
Remove yet another debug print, run tests again
e06766a
@leftwo
Remove another comment, run tests again
7871810
@leftwo
Update comment, run tests again
687fe86
@leftwo
fix mismerge for mgd_checksums
f4b3271
@leftwo
remove debug function, run tests again
725da36
@leftwo leftwo changed the title WIP test flake debugging Reduce test flakes Sep 8, 2025
@leftwo leftwo requested a review from jmpesp September 8, 2025 17:56
@leftwo leftwo marked this pull request as ready for review September 8, 2025 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jmpesp jmpesp Awaiting requested review from jmpesp

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@leftwo @Nieuwejaar @KaiSforza @JustinAzoff @david-crespo @rcgoodfellow