[DO NOT MERGE] test(visual): improve stability pt 2 #1995
+27
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
✅ Deploy Preview for stacks ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm trying to validate (or invalidate) some theories on what tends to cause seemingly random visual regression test failures. If I narrow it down, expect this PR to become ready for review. For now, no need to review or merge.
Visual regression test runs
I tried running the tests locally to gauge flaky-ness. Out of 4 runs each, this branch had no failures while
develophad 2. Small sample size but it's promising 🤞Note
The runs for this branch do take ~10% longer. I think that's a fine trade-off if we gain stability. That said, I'd like to pick through these changes and see if I can narrow down what helps and what doesn't.
develop
~/Code/Stacks develop 05:43:53 PM
❯ time npm run test:visual -w packages/stacks-classic; tput bel
Building docker image...
Running docker container...
lib/components/activity-indicator/activity-indicator.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/anchor/anchor.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/avatar/avatar.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/award-bling/award-bling.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/badge/badge.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/banner/banner.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/block-link/block-link.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
Chromium: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Firefox: |██████████████████████████████| 42/42 test files | 2636 passed, 0 failed
Webkit: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Error while running tests.
Exited with status code 1
npm error Lifecycle script
test:visual:cifailed with error:npm error code 1
npm error path /Users/dcormier/Code/Stacks/packages/stacks-classic
npm error workspace @stackoverflow/[email protected]
npm error location /Users/dcormier/Code/Stacks/packages/stacks-classic
npm error command failed
npm error command sh -c pwsh run-test-visual.ps1 npx web-test-runner --group=visual --config ./visual-runner/stacks-classic-runner-config/web-test-runner.config.mjs
npm error Lifecycle script
test:visualfailed with error:npm error code 1
npm error path /Users/dcormier/Code/Stacks/packages/stacks-classic
npm error workspace @stackoverflow/[email protected]
npm error location /Users/dcormier/Code/Stacks/packages/stacks-classic
npm error command failed
npm error command sh -c npm run test:visual:ci -- --config ./visual-runner/stacks-classic-runner-config/web-test-runner.config.mjs
npm run test:visual -w packages/stacks-classic 1.41s user 1.88s system 1% cpu 2:50.45 total
~/Code/Stacks develop 2m 50s 05:46:47 PM
❯ time npm run test:visual -w packages/stacks-classic; tput bel
Building docker image...
Running docker container...
Chromium: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Firefox: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Webkit: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Finished running tests in 146.4s, all tests passed! 🎉
Exited with status code 0
npm run test:visual -w packages/stacks-classic 1.07s user 1.80s system 1% cpu 2:31.52 total
~/Code/Stacks develop 06:09:11 PM
❯ time npm run test:visual -w packages/stacks-classic; tput bel
Building docker image...
Running docker container...
lib/components/activity-indicator/activity-indicator.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/anchor/anchor.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/avatar/avatar.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/award-bling/award-bling.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/badge/badge.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/banner/banner.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
lib/components/breadcrumbs/breadcrumbs.visual.test.ts:
❌ The browser was unable to create and start a test page after 60000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox)
Chromium: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Firefox: |██████████████████████████████| 42/42 test files | 2660 passed, 0 failed
Webkit: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Error while running tests.
Exited with status code 1
npm error Lifecycle script
test:visual:cifailed with error:npm error code 1
npm error path /Users/dcormier/Code/Stacks/packages/stacks-classic
npm error workspace @stackoverflow/[email protected]
npm error location /Users/dcormier/Code/Stacks/packages/stacks-classic
npm error command failed
npm error command sh -c pwsh run-test-visual.ps1 npx web-test-runner --group=visual --config ./visual-runner/stacks-classic-runner-config/web-test-runner.config.mjs
npm error Lifecycle script
test:visualfailed with error:npm error code 1
npm error path /Users/dcormier/Code/Stacks/packages/stacks-classic
npm error workspace @stackoverflow/[email protected]
npm error location /Users/dcormier/Code/Stacks/packages/stacks-classic
npm error command failed
npm error command sh -c npm run test:visual:ci -- --config ./visual-runner/stacks-classic-runner-config/web-test-runner.config.mjs
npm run test:visual -w packages/stacks-classic 1.46s user 1.84s system 1% cpu 2:52.82 total
This branch
~/Code/Stacks SPARK-52/improve-visual-regression-test-stability 05:39:11 PM
❯ time npm run test:visual -w packages/stacks-classic; tput bel
Building docker image...
Running docker container...
Chromium: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Firefox: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Webkit: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Finished running tests in 181.4s, all tests passed! 🎉
Exited with status code 0
npm run test:visual -w packages/stacks-classic 1.40s user 2.20s system 1% cpu 3:14.30 total
~/Code/Stacks SPARK-52/improve-visual-regression-test-stability 05:55:08 PM
❯ time npm run test:visual -w packages/stacks-classic; tput bel
Building docker image...
Running docker container...
Chromium: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Firefox: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Webkit: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Finished running tests in 179.4s, all tests passed! 🎉
Exited with status code 0
npm run test:visual -w packages/stacks-classic 1.06s user 1.82s system 1% cpu 3:04.52 total
~/Code/Stacks SPARK-52/improve-visual-regression-test-stability 3m 5s 05:58:23 PM
❯ time npm run test:visual -w packages/stacks-classic; tput bel
Building docker image...
Running docker container...
Chromium: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Firefox: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Webkit: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Finished running tests in 180.1s, all tests passed! 🎉
Exited with status code 0
npm run test:visual -w packages/stacks-classic 1.09s user 1.88s system 1% cpu 3:05.48 total
~/Code/Stacks SPARK-52/improve-visual-regression-test-stability 06:13:59 PM
❯ time npm run test:visual -w packages/stacks-classic; tput bel
Building docker image...
Running docker container...
Chromium: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Firefox: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Webkit: |██████████████████████████████| 42/42 test files | 3028 passed, 0 failed
Finished running tests in 181.7s, all tests passed! 🎉
Exited with status code 0
npm run test:visual -w packages/stacks-classic 1.03s user 1.89s system 1% cpu 3:07.30 total
Update Fri Oct 3, 2025 12:42pm ET: Two visual regression test workflow runs for this PR and still passing 😎
Update Fri Oct 3, 2025 2:11pm ET: Make that three.