[E2E][CI] Ensure version coverage for WooCommerce and PHP #11035
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mgascam
changed the title
Test the buildOption 1. Jetpack Beta
Option 2. Jurassic Ninja - available for logged-in A12s🚀 Launch a JN site with this branch 🚀 ℹ️ Install this Tampermonkey script to get more options. Build info:
Note: the build is updated when a new commit is pushed to this PR. |
|
Size Change: 0 B Total Size: 870 kB ℹ️ View Unchanged
|
6 tasks
There was a problem hiding this comment.
Pull Request Overview
This PR implements comprehensive version coverage for E2E tests by introducing dynamic matrix generation with an L-1 policy, expanded PHP version testing, and improved retry logic. The changes ensure better coverage of WooCommerce and PHP versions while optimizing CI execution time.
- Dynamic version resolution using a new script that automatically fetches WC versions and applies L-1 policy
- Enhanced PHP version coverage (7.3, 8.3, 8.4) with strategic assignment to different WC versions
- Improved retry logic that selectively re-runs only failed spec files instead of using Playwright's built-in retries
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/e2e-test.yml |
Updated main E2E workflow to use dynamic matrix generation with optimized PHP/WC version combinations |
.github/workflows/e2e-pull-request.yml |
Enhanced PR workflow to test against both L-1 and latest WC versions with PHP 8.3 |
.github/scripts/generate-wc-matrix.sh |
New script for dynamic WC version resolution with L-1 policy implementation |
.github/scripts/README.md |
Comprehensive documentation for the new matrix generation strategy |
.github/actions/setup-php/action.yml |
Added support for E2E_PHP_VERSION environment variable override |
.github/actions/e2e/run-log-tests/action.yml |
Improved retry logic to selectively re-run failed spec files |
tests/e2e/playwright.config.ts |
Disabled built-in retries in favor of custom retry strategy |
tests/e2e/env/setup.sh |
Removed legacy beta/RC version resolution logic |
tests/e2e/README.md |
Updated documentation with new retry strategy and matrix generation details |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
htdat
approved these changes
Sep 5, 2025
There was a problem hiding this comment.
Code changes resolve the coverage issue, and retries work as I double checked at https://github.com/Automattic/woocommerce-payments/actions/runs/17467393007/job/49619897091?pr=11035
Overall, LGTM with some improvement comments!
htdat
reviewed
Sep 5, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #WOOPMNT-5249
Changes proposed in this Pull Request
Split from #11002 to focus on the following changes (see comment):
Warning
This PR hinders the stability of the E2E test runs in CI due to test flakiness, but I'm fixing those in a separate PR, to reduce the scope of changes and facilitate reviews
Details
After reviewing the state of E2E testing (pcreKM-3yi-p2) we identified some version coverage gaps:
This change aims to cover those gaps by introducing new test jobs, while keeping in mind the impact to the time it takes to run the workflows.
PHP versions
The current E2E testing strategy is updated to address PHP version coverage gaps:
Please note that we won't be testing each WC version using every PHP version because that would introduce too many jobs, hindering the time it takes to run them. Instead I'm using a "smart" approach to keep the number of jobs to a minimal without sacrificing too much testing coverage.
WooCommerce versions
I'm adding new jobs to test with L-1 WooCommerce version, to align with this recent announcement pcShBQ-3pc-p2#comment-5118. This is the only change affecting the WC versions used in the tests.
About "beta" and "rc" versions
Please note that "beta" and "rc" versions are only included in the matrix when available. While working with the script to fetch the versions from
https://api.wordpress.org/plugins/info/1.0/woocommerce.jsonI noticed there is no "beta" version currently available for the next version 10.1.0. I believe this is expected but thought it was worth mentioning it. Also I realized that the current workflow is using a 9.9.0-beta.1 but I think the correct beta to use should be 10.1.0-beta.x (if it exists). This inconsistency would be fixed by this PR.Job matrix comparison
Changes proposed in this Pull Request
.github/workflows/e2e-pull-request.yml: Added L-1 testing to PR workflow and include PHP in matrix..github/actions/setup-php/action.yml: supportE2E_PHP_VERSIONoverride..github/scripts/generate-wc-matrix.sh: New dynamic version resolution script. This is heavily inspired by @marcinbot's E2E - fix pulling non-latest WC versions, set up tests with rc #10990.github/workflows/e2e-test.yml: The main branch workflow is updated to:Testing instructions
npm run changelogto add a changelog file, choosepatchto leave it empty if the change is not significant. You can add multiple changelog files in one PR by running this command a few times.Post merge