Shard Update Test Expectations PR #6100
Open
+179
−28
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🎭 Playwright Test Results❌ Some tests failed ⏰ Completed at: 10/21/2025, 08:03:56 AM UTC 📈 Summary
📊 Test Reports by Browser
🎉 Click on the links above to view detailed test results for each browser configuration. |
🎨 Storybook Build Status✅ Build completed successfully! ⏰ Completed at: 10/21/2025, 07:44:30 AM UTC 🔗 Links🎉 Your Storybook is ready for review! |
dosubot
bot
added
the
size:L
|
will be nice to have both tho |
…aching and sharded snapshot handling
DrJKL
force-pushed
the
shard-update-expectations
branch
from
e103b7e to
d0c44e8
Compare
Bundle Size ReportSummary
Category Glance Per-category breakdownApp Entry Points — 3.31 MB (baseline 3.31 MB) • ⚪ 0 B_Main entry bundles and manifests_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ---------------------------------------- | ------- | ------- | ----------------------- | ---------------------- | ----------------------- | | ~~assets/index-CM0W3WRg.js~~ _(removed)_ | 2.69 MB | — | 🟢 -2.69 MB | 🟢 -561 kB | 🟢 -424 kB | | **assets/index-De9LMPsl.js** _(new)_ | — | 2.69 MB | 🔴 +2.69 MB | 🔴 +561 kB | 🔴 +424 kB | | ~~assets/index-BB9uj6Rd.js~~ _(removed)_ | 616 kB | — | 🟢 -616 kB | 🟢 -114 kB | 🟢 -90.4 kB | | **assets/index-DNWfg-oG.js** _(new)_ | — | 616 kB | 🔴 +616 kB | 🔴 +114 kB | 🔴 +90.3 kB |Status: 2 added / 2 removed Graph Workspace — 708 kB (baseline 708 kB) • 🟢 -1 B_Graph editor runtime, canvas, workflow orchestration_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | -------------------------------------------- | ------ | ------ | ---------------------- | ---------------------- | ---------------------- | | ~~assets/GraphView-BiLfiv4R.js~~ _(removed)_ | 708 kB | — | 🟢 -708 kB | 🟢 -139 kB | 🟢 -107 kB | | **assets/GraphView-CX0LMhyh.js** _(new)_ | — | 708 kB | 🔴 +708 kB | 🔴 +139 kB | 🔴 +107 kB |Status: 1 added / 1 removed Views & Navigation — 8.15 kB (baseline 8.15 kB) • ⚪ 0 B_Top-level views, pages, and routed surfaces_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ------------------------------------------------- | ------- | ------- | ----------------------- | ----------------------- | ----------------------- | | ~~assets/UserSelectView-AoFyDL04.js~~ _(removed)_ | 8.15 kB | — | 🟢 -8.15 kB | 🟢 -2.46 kB | 🟢 -2.16 kB | | **assets/UserSelectView-DwvA16Ep.js** _(new)_ | — | 8.15 kB | 🔴 +8.15 kB | 🔴 +2.46 kB | 🔴 +2.16 kB |Status: 1 added / 1 removed Panels & Settings — 294 kB (baseline 294 kB) • ⚪ 0 B_Configuration panels, inspectors, and settings screens_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ---------------------------------------------------- | ------- | ------- | ----------------------- | ----------------------- | ----------------------- | | ~~assets/CreditsPanel-ff8q5QmJ.js~~ _(removed)_ | 22.1 kB | — | 🟢 -22.1 kB | 🟢 -5.28 kB | 🟢 -4.62 kB | | **assets/CreditsPanel-HQgC6tEE.js** _(new)_ | — | 22.1 kB | 🔴 +22.1 kB | 🔴 +5.28 kB | 🔴 +4.62 kB | | **assets/KeybindingPanel-BQ28izky.js** _(new)_ | — | 15.2 kB | 🔴 +15.2 kB | 🔴 +3.76 kB | 🔴 +3.31 kB | | ~~assets/KeybindingPanel-CjcXLNwN.js~~ _(removed)_ | 15.2 kB | — | 🟢 -15.2 kB | 🟢 -3.76 kB | 🟢 -3.32 kB | | **assets/ExtensionPanel-BjMA0UTg.js** _(new)_ | — | 12.1 kB | 🔴 +12.1 kB | 🔴 +2.82 kB | 🔴 +2.48 kB | | ~~assets/ExtensionPanel-BRbIXTzI.js~~ _(removed)_ | 12.1 kB | — | 🟢 -12.1 kB | 🟢 -2.83 kB | 🟢 -2.47 kB | | ~~assets/AboutPanel-5RzAul56.js~~ _(removed)_ | 10.3 kB | — | 🟢 -10.3 kB | 🟢 -2.66 kB | 🟢 -2.33 kB | | **assets/AboutPanel-jAK3-5Q6.js** _(new)_ | — | 10.3 kB | 🔴 +10.3 kB | 🔴 +2.66 kB | 🔴 +2.33 kB | | ~~assets/ServerConfigPanel-B065dJLQ.js~~ _(removed)_ | 8.2 kB | — | 🟢 -8.2 kB | 🟢 -2.16 kB | 🟢 -1.9 kB | | **assets/ServerConfigPanel-eiRpxNaj.js** _(new)_ | — | 8.2 kB | 🔴 +8.2 kB | 🔴 +2.16 kB | 🔴 +1.9 kB | | **assets/UserPanel-BEni4not.js** _(new)_ | — | 7.91 kB | 🔴 +7.91 kB | 🔴 +2.05 kB | 🔴 +1.79 kB | | ~~assets/UserPanel-D_NmyzPB.js~~ _(removed)_ | 7.91 kB | — | 🟢 -7.91 kB | 🟢 -2.06 kB | 🟢 -1.79 kB | | assets/settings-B-df0dZe.js | 20.7 kB | 20.7 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-CI6OKvJn.js | 22.9 kB | 22.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-CXGVj_nD.js | 24.5 kB | 24.5 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-DfQ6dSJj.js | 31.6 kB | 31.6 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-DJ2QgDzm.js | 25.2 kB | 25.2 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-DRNLPMG6.js | 23.7 kB | 23.7 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-DVVycxDc.js | 19.9 kB | 19.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-G6Dybj1b.js | 24.1 kB | 24.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-M6_GZccG.js | 26 kB | 26 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |Status: 6 added / 6 removed UI Components — 12.3 kB (baseline 12.3 kB) • ⚪ 0 B_Reusable component library chunks_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ----------------------------------------------------------------- | ------- | ------- | ----------------------- | ----------------------- | ----------------------- | | **assets/ComfyQueueButton-7nxH93_1.js** _(new)_ | — | 11.1 kB | 🔴 +11.1 kB | 🔴 +2.76 kB | 🔴 +2.44 kB | | ~~assets/ComfyQueueButton-DJPOXiVu.js~~ _(removed)_ | 11.1 kB | — | 🟢 -11.1 kB | 🟢 -2.76 kB | 🟢 -2.44 kB | | assets/UserAvatar.vue_vue_type_script_setup_true_lang-C9bSkTC5.js | 1.12 kB | 1.12 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |Status: 1 added / 1 removed Data & Services — 10 kB (baseline 10 kB) • ⚪ 0 B_Stores, services, APIs, and repositories_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ---------------------------------------------------- | ------- | ------- | ----------------------- | ----------------------- | ----------------------- | | ~~assets/keybindingService-Cpmp8sEP.js~~ _(removed)_ | 7.21 kB | — | 🟢 -7.21 kB | 🟢 -1.75 kB | 🟢 -1.51 kB | | **assets/keybindingService-X-5rFS-f.js** _(new)_ | — | 7.21 kB | 🔴 +7.21 kB | 🔴 +1.75 kB | 🔴 +1.5 kB | | assets/serverConfigStore-B1bw4Pe0.js | 2.79 kB | 2.79 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |Status: 1 added / 1 removed Utilities & Hooks — 1.07 kB (baseline 1.07 kB) • ⚪ 0 B_Helpers, composables, and utility bundles_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/mathUtil-CTARWQ-l.js | 1.07 kB | 1.07 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |Vendor & Third-Party — 5.36 MB (baseline 5.36 MB) • ⚪ 0 B_External libraries and shared vendor chunks_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/vendor-other-kaNE-JGc.js | 3.22 MB | 3.22 MB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/vendor-primevue-PESgPnbc.js | 517 B | 517 B | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/vendor-tiptap-DKA7Hxfn.js | 232 kB | 232 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/vendor-visualization-BEfdbjRw.js | 1.82 MB | 1.82 MB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/vendor-vue-QImF2beP.js | 92.4 kB | 92.4 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |Other — 2.79 MB (baseline 2.79 MB) • ⚪ 0 B_Bundles that do not match a named category_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/commands-B2KZRBmX.js | 15.1 kB | 15.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-Bw-ckyga.js | 13.9 kB | 13.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-C_NmM85I.js | 13.8 kB | 13.8 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-CuozCW4W.js | 14 kB | 14 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-DGfVUJCR.js | 16.2 kB | 16.2 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-dOJNDogK.js | 14.5 kB | 14.5 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-DwiE551e.js | 14.7 kB | 14.7 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-Fw7mvqSy.js | 13.1 kB | 13.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-FXnO1W4Q.js | 13.2 kB | 13.2 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-6UgCUkrV.js | 108 kB | 108 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-BfHN1fzx.js | 125 kB | 125 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-BhulUfFD.js | 77.5 kB | 77.5 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-BPHe683n.js | 92.4 kB | 92.4 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-C75C4LWt.js | 90.9 kB | 90.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-CEhf19j2.js | 99.4 kB | 99.4 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-CipazGd8.js | 79.3 kB | 79.3 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-CKz_lTAz.js | 94.3 kB | 94.3 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-Dzm38Va4.js | 90.3 kB | 90.3 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-BePSqkA4.js | 195 kB | 195 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-BfT7dJcF.js | 204 kB | 204 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-BiAtoiXc.js | 194 kB | 194 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-CDfbduPY.js | 219 kB | 219 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-CDurg_KW.js | 197 kB | 197 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-CE-vG3RG.js | 182 kB | 182 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-DAwVV156.js | 200 kB | 200 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-DexhCMEi.js | 233 kB | 233 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-kTrYLFPK.js | 184 kB | 184 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | |
|
[claude-review] ## PR Review: Shard Update Test Expectations ✅ Job Rename Safety Check (test → setup)The rename from
The rename better reflects the job's purpose (setup + caching) vs. the old misleading 💡 Testing RecommendationStrongly recommend testing before merge. Here's how: Option 1: Enable on Current Branch (Recommended)Add temporary push trigger to test the full workflow: on:
push:
branches: [shard-update-expectations] # Test on this PR branch
pull_request:
types: [labeled]
issue_comment:
types: [created]Testing approach:
Option 2: Manual Workflow DispatchAdd on:
workflow_dispatch:
inputs:
pr_number:
description: 'PR number to test'
required: true
pull_request:
types: [labeled]
issue_comment:
types: [created]🔍 Code ReviewArchitecture: ✅ Excellent refactoring
Key Improvements:
Potential Issues:
📋 Summary
Recommendation: Add temporary push trigger to test on this branch before merging to main. The refactoring looks solid but the sharding + caching + merge logic is complex enough to warrant validation. |
DrJKL
reviewed
Oct 20, 2025
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Comprehensive PR Review
This review is generated by Claude. It may not always be accurate, as with human reviewers. If you believe that any of the comments are invalid or incorrect, please state why for each. For others, please implement the changes in one way or another.
Review Summary
PR: Shard Update Test Expectations PR (#6100)
Impact: 120 additions, 27 deletions across 1 file
Issue Distribution
- Critical: 0
- High: 3
- Medium: 4
- Low: 1
Category Breakdown
- Architecture: 2 issues
- Security: 1 issue
- Performance: 2 issues
- Code Quality: 3 issues
Key Findings
Architecture & Design
The workflow refactoring introduces significant improvements by sharding Playwright test execution, but several architectural concerns need attention:
- Caching Strategy: The current approach caches the entire workspace (path: .) which is inefficient and potentially risky. This includes .git directories, temporary files, and artifacts that shouldn't be cached.
- Job Dependencies: Using 'if: always()' on the merge-and-commit job bypasses important failure detection, potentially leading to commits of partial or corrupted snapshot data.
Security Considerations
- CI/CD Pipeline Security: Hard-coded sleep delays in CI pipelines are anti-patterns that can mask race conditions and create unnecessary attack surfaces.
Performance Impact
- Cache Inefficiency: The timestamp-based cache key means cache will never be hit, defeating the purpose and wasting CI resources.
- Artifact Management: Wildcard patterns may upload unnecessary data, increasing storage costs and transfer times.
Integration Points
The sharding approach is sound architecturally, but needs better error handling and validation to ensure reliable snapshot merging across shards.
Positive Observations
- Excellent Sharding Implementation: The 4-shard parallelization will significantly improve test execution time
- Improved PR Context Handling: Better extraction and passing of PR metadata between jobs
- Job Separation: Clean separation of setup, execution, and merge phases follows good CI/CD practices
- Artifact Management: Proper use of GitHub Actions artifacts for inter-job communication
- Good Error Tolerance: 'continue-on-error: true' allows partial success scenarios
References
Next Steps
- Address high-priority caching and error handling issues before merge
- Consider implementing proper cache key strategy for performance gains
- Add validation checks for artifact integrity
- Test the workflow thoroughly with various failure scenarios
This is a comprehensive automated review. For architectural decisions requiring human judgment, please request additional manual review.
github-actions
bot
removed
the
claude-review
DrJKL
reviewed
Oct 21, 2025
| description: 'Whether to launch the server after setup' | ||
| required: false | ||
| default: 'false' | ||
| skip_checkout: |
There was a problem hiding this comment.
This feels wrong. What's left at that point?
There was a problem hiding this comment.
Its for the cache though
DrJKL
reviewed
Oct 21, 2025
Comment on lines
+67
to
+68
| # Save expensive build artifacts (Python env, built frontend, node_modules) | ||
| # Source code will be checked out fresh in sharded jobs |
There was a problem hiding this comment.
If you look at @snomiao's recent cache removal change, it can sometimes be more expensive to try to save time like this.
DrJKL
reviewed
Oct 21, 2025
| path: | | ||
| ComfyUI | ||
| dist | ||
| node_modules |
There was a problem hiding this comment.
If you're using the built files, do you need node_modules?
And ComfyUI is just the checked out Comfy Repo, which might be faster to checkout than to restore from cache.
DrJKL
reviewed
Oct 21, 2025
Comment on lines
+169
to
+173
| # Verify target directory exists | ||
| if [ ! -d "browser_tests" ]; then | ||
| echo "::error::Target directory 'browser_tests' does not exist" | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
This might be a little too far. It's a lot to set as an inline script. I was more thinking of just letting the stderr log so that we could see it in debug logs, not trying to handle every contingency.
DrJKL
reviewed
Oct 21, 2025
There was a problem hiding this comment.
wanted to test it
will take it out before merging
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request significantly refactors the Playwright expectations update workflow to improve reliability, efficiency, and maintainability. The workflow is now split into three coordinated jobs—setup, sharded snapshot updates, and merge/commit—enabling parallel test execution and artifact management. Key improvements include sharding Playwright snapshot updates, robust caching and artifact handling, and more reliable PR context handling.
Workflow Restructuring and Sharding:
setup(prepares environment and caches it),update-snapshots-sharded(runs Playwright snapshot updates in four parallel shards), andmerge-and-commit(merges results and commits updates). This enables faster, more reliable snapshot updates. [1] [2]Caching and Artifact Management:
Improved PR Context Handling:
Job and Step Renaming/Cleanup:
testtosetup, and redundant or unnecessary steps (such as the old branch SHA extraction) are removed for clarity and maintainability. [1] [2]Comment and Label Automation Improvements:
┆Issue is synchronized with this Notion page by Unito