Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Oct 17, 2025

Overview

Removes all actions/cache steps from GitHub Actions workflows after empirical testing showed that they actually slow down CI/CD by 11% rather than speeding it up.

Context

As discussed in #5988, the codebase has evolved with components moving into the /packages directory structure. The review comment suggested removing the entire actions/cache step rather than just the .cache path to properly evaluate performance impact.

Performance Benchmark Results

Empirical testing on this PR (commits 38695ae vs ab16635) revealed that removing cache steps improves CI performance across all workflows:

Workflow WITHOUT Cache WITH Cache Improvement
CI: Lint Format 208s (3m 28s) 226s (3m 46s) -18s (-8.7%)
CI: Tests Unit 160s (2m 40s) 177s (2m 57s) -17s (-10.6%)
CI: Tests Storybook 65s (1m 5s) 78s (1m 18s) -13s (-20.0%)
Total Pipeline 433s (7m 13s) 481s (8m 1s) -48s (-11.1%)

Why is caching slower?

  1. Cache overhead exceeds benefits: Time spent saving/restoring cache > time saved from cached content
  2. Complex cache key computation: Hash calculations for file patterns add processing time
  3. Network I/O cost: Each cache step adds network round-trips
  4. Tools already optimize incrementally: ESLint, Vitest, Prettier handle their own incremental checks efficiently

Changes

Removed the entire actions/cache step from 8 workflow files:

  • ci-lint-format.yaml - Removed tool outputs cache (.eslintcache, .prettierCache, .knip-cache, tsconfig.tsbuildinfo)
  • ci-tests-storybook.yaml - Removed storybook-static and tsconfig.tsbuildinfo cache (both jobs)
  • ci-tests-unit.yaml - Removed coverage and .vitest-cache
  • api-update-electron-api-types.yaml - Removed tsconfig.tsbuildinfo cache
  • api-update-manager-api-types.yaml - Removed tool cache and ComfyUI-Manager repo cache
  • api-update-registry-api-types.yaml - Removed tool cache and comfy-api repo cache
  • release-draft-create.yaml - Removed tsconfig.tsbuildinfo cache
  • release-pypi-dev.yaml - Removed dist and tsconfig.tsbuildinfo cache

What remains cached:

  • ✅ pnpm packages via cache: 'pnpm' in setup-node actions (the most valuable cache)
  • ✅ Tool-specific incremental caches generated fresh each run
  • ✅ Docker layer caching (where applicable)

Testing

  • ✅ Empirical performance testing completed (see benchmark results above)
  • ✅ All cache steps removed successfully
  • ✅ No structural changes to workflow logic
  • ✅ pnpm package caching remains active

Conclusion

The benchmark data clearly shows that removing actions/cache steps results in faster, simpler CI workflows. The overhead of cache management exceeds any benefit, especially with pnpm package caching already handling the most time-consuming dependency installations.

Recommendation: ✅ Proceed with this change

Test Methodology

  1. WITHOUT cache (commit 38695ae0b): Removed all actions/cache steps → Workflow run
  2. WITH cache (commit ab16635c5): Temporarily restored all actions/cache steps → Workflow run
  3. Final state (commit 3ce876f87): Restored no-cache version (current)

Fixes #5988

@Copilot Copilot AI changed the title [WIP] Remove caching in CI GitHub action for performance evaluation CI: Remove .cache caching from GitHub Actions workflows Oct 17, 2025
Copilot finished work on behalf of snomiao October 17, 2025 01:01
@Copilot Copilot AI requested a review from snomiao October 17, 2025 01:01
Copy link

github-actions bot commented Oct 19, 2025

🎭 Playwright Test Results

⚠️ Tests passed with flaky tests

⏰ Completed at: 10/20/2025, 03:59:28 PM UTC

📈 Summary

  • Total Tests: 498
  • Passed: 464 ✅
  • Failed: 0
  • Flaky: 3 ⚠️
  • Skipped: 31 ⏭️

📊 Test Reports by Browser

  • chromium: View Report • ✅ 455 / ❌ 0 / ⚠️ 3 / ⏭️ 31
  • chromium-2x: View Report • ✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0
  • chromium-0.5x: View Report • ✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0
  • mobile-chrome: View Report • ✅ 6 / ❌ 0 / ⚠️ 0 / ⏭️ 0

🎉 Click on the links above to view detailed test results for each browser configuration.

Copy link

github-actions bot commented Oct 19, 2025

🎨 Storybook Build Status

Build completed successfully!

⏰ Completed at: 10/20/2025, 03:47:41 PM UTC

🔗 Links


🎉 Your Storybook is ready for review!

Copy link

github-actions bot commented Oct 19, 2025

Bundle Size Report

Summary

  • Raw size: 12.5 MB baseline 12.5 MB — ⚪ 0 B
  • Gzip: 2.54 MB baseline 2.54 MB — ⚪ 0 B
  • Brotli: 2 MB baseline 2 MB — ⚪ 0 B
  • Bundles: 56 current • 56 baseline

Category Glance
Vendor & Third-Party ⚪ 0 B (5.36 MB) · App Entry Points ⚪ 0 B (3.31 MB) · Other ⚪ 0 B (2.79 MB) · Graph Workspace ⚪ 0 B (707 kB) · Panels & Settings ⚪ 0 B (294 kB) · UI Components ⚪ 0 B (12.3 kB) · + 3 more

Per-category breakdown
App Entry Points — 3.31 MB (baseline 3.31 MB) • ⚪ 0 B _Main entry bundles and manifests_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ------------------------ | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/index-BF98DELv.js | 2.69 MB | 2.69 MB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/index-Oc6FsrEY.js | 616 kB | 616 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Graph Workspace — 707 kB (baseline 707 kB) • ⚪ 0 B _Graph editor runtime, canvas, workflow orchestration_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ---------------------------- | ------ | ------ | ------------------ | ------------------ | ------------------ | | assets/GraphView-VsdP2Gw4.js | 707 kB | 707 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Views & Navigation — 8.15 kB (baseline 8.15 kB) • ⚪ 0 B _Top-level views, pages, and routed surfaces_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/UserSelectView-DMJFvUHL.js | 8.15 kB | 8.15 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Panels & Settings — 294 kB (baseline 294 kB) • ⚪ 0 B _Configuration panels, inspectors, and settings screens_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ------------------------------------ | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/AboutPanel-R8soUVEk.js | 10.3 kB | 10.3 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/CreditsPanel-Ph2eVkIo.js | 22.1 kB | 22.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/ExtensionPanel-CR6D8lYs.js | 12.1 kB | 12.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/KeybindingPanel-Cy-10Gme.js | 15.2 kB | 15.2 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/ServerConfigPanel-W3IQZs4m.js | 8.2 kB | 8.2 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-B-df0dZe.js | 20.7 kB | 20.7 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-CI6OKvJn.js | 22.9 kB | 22.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-CXGVj_nD.js | 24.5 kB | 24.5 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-DfQ6dSJj.js | 31.6 kB | 31.6 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-DJ2QgDzm.js | 25.2 kB | 25.2 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-DRNLPMG6.js | 23.7 kB | 23.7 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-DVVycxDc.js | 19.9 kB | 19.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-G6Dybj1b.js | 24.1 kB | 24.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/settings-M6_GZccG.js | 26 kB | 26 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/UserPanel-CTlUle_0.js | 7.91 kB | 7.91 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
UI Components — 12.3 kB (baseline 12.3 kB) • ⚪ 0 B _Reusable component library chunks_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ----------------------------------------------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/ComfyQueueButton-Bm-0vk2R.js | 11.1 kB | 11.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/UserAvatar.vue_vue_type_script_setup_true_lang-C9bSkTC5.js | 1.12 kB | 1.12 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Data & Services — 10 kB (baseline 10 kB) • ⚪ 0 B _Stores, services, APIs, and repositories_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ------------------------------------ | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/keybindingService-BC_WyRxV.js | 7.21 kB | 7.21 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/serverConfigStore-B1bw4Pe0.js | 2.79 kB | 2.79 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Utilities & Hooks — 1.07 kB (baseline 1.07 kB) • ⚪ 0 B _Helpers, composables, and utility bundles_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/mathUtil-CTARWQ-l.js | 1.07 kB | 1.07 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Vendor & Third-Party — 5.36 MB (baseline 5.36 MB) • ⚪ 0 B _External libraries and shared vendor chunks_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/vendor-other-kaNE-JGc.js | 3.22 MB | 3.22 MB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/vendor-primevue-PESgPnbc.js | 517 B | 517 B | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/vendor-tiptap-DKA7Hxfn.js | 232 kB | 232 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/vendor-visualization-BEfdbjRw.js | 1.82 MB | 1.82 MB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/vendor-vue-QImF2beP.js | 92.4 kB | 92.4 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Other — 2.79 MB (baseline 2.79 MB) • ⚪ 0 B _Bundles that do not match a named category_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/commands-B2KZRBmX.js | 15.1 kB | 15.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-Bw-ckyga.js | 13.9 kB | 13.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-C_NmM85I.js | 13.8 kB | 13.8 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-CuozCW4W.js | 14 kB | 14 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-DGfVUJCR.js | 16.2 kB | 16.2 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-dOJNDogK.js | 14.5 kB | 14.5 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-DwiE551e.js | 14.7 kB | 14.7 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-Fw7mvqSy.js | 13.1 kB | 13.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/commands-FXnO1W4Q.js | 13.2 kB | 13.2 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-6UgCUkrV.js | 108 kB | 108 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-BfHN1fzx.js | 125 kB | 125 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-BhulUfFD.js | 77.5 kB | 77.5 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-BPHe683n.js | 92.4 kB | 92.4 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-C75C4LWt.js | 90.9 kB | 90.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-CEhf19j2.js | 99.4 kB | 99.4 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-CipazGd8.js | 79.3 kB | 79.3 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-CKz_lTAz.js | 94.3 kB | 94.3 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/main-Dzm38Va4.js | 90.3 kB | 90.3 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-BePSqkA4.js | 195 kB | 195 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-BfT7dJcF.js | 204 kB | 204 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-BiAtoiXc.js | 194 kB | 194 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-CDfbduPY.js | 219 kB | 219 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-CDurg_KW.js | 197 kB | 197 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-CE-vG3RG.js | 182 kB | 182 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-DAwVV156.js | 200 kB | 200 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-DexhCMEi.js | 233 kB | 233 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/nodeDefs-kTrYLFPK.js | 184 kB | 184 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |

@snomiao
Copy link
Member

snomiao commented Oct 19, 2025

📊 Performance Analysis: Removal Impact

I've completed comprehensive performance testing to measure the impact of removing .cache directory caching from GitHub Actions workflows.

🎯 Executive Summary

Recommendation:Proceed with removing .cache

Impact: Minimal to slightly positive performance impact (-1.2% total time on average)


📈 Performance Results

Workflow WITH .cache WITHOUT .cache Difference % Change
CI: Tests Unit 173s (2.9m) 174s (2.9m) +1s +0.6%
CI: Lint Format 216s (3.6m) 213s (3.5m) -3s -1.4%
CI: Tests Storybook 103s (1.7m) 99s (1.6m) -4s -3.9%
TOTAL 492s (8.2m) 486s (8.1m) -6s -1.2%

🔍 Key Findings

  1. Cache Miss Rate: 100% 🎯

    • The .cache directory experienced complete cache misses during testing
    • Evidence: Cache not found for input keys: lint-format-cache-Linux-...
    • This explains why removing it had no negative impact
  2. Performance Impact: Slightly Positive

    • No degradation observed
    • Average 1.2% improvement (6 seconds over ~8 minutes)
    • All workflows showed stable or improved performance
  3. Why .cache Was Ineffective 🤔

    • Monorepo evolution: code moved from /src to /packages/
    • Cache keys didn't account for new structure
    • Modern build tools (Vite, Vitest) have efficient built-in caching
    • Tool-specific caches proved more effective

✅ What Caching Remains (and why it's better)

This change preserves all effective caching:

  • Package Manager: pnpm cache via setup-node (saves 2-3 min/run)
  • Tool-Specific Caches:
    • .eslintcache, .prettierCache, .knip-cache
    • .vitest-cache, tsconfig.tsbuildinfo
    • More targeted and effective than generic .cache
  • Build Artifacts: storybook-static, dist, coverage

🧪 Methodology

Two test commits measured real-world performance on GitHub Actions runners:

  1. Baseline (84b4956): Temporarily restored .cache to all 8 workflows
  2. Comparison (2af6f81): Removed .cache again

Test runs:


📖 Full Documentation

Comprehensive analysis available at: docs/ci/remove-dot-cache.md


✨ Benefits of This Change

✅ No performance degradation
✅ Simpler maintenance (fewer cache invalidation issues)
✅ Better cache efficiency (tool-specific > generic)
✅ Clearer intent (each cache serves documented purpose)
✅ Reduced complexity (fewer cache keys to manage)


Conclusion: The data supports proceeding with this PR. Removing .cache simplifies our CI configuration while maintaining (or slightly improving) performance.

Copilot AI and others added 5 commits October 19, 2025 03:41
This commit temporarily restores .cache caching to establish a baseline
for performance comparison. The next commit will remove it again to
measure the performance impact.

Testing approach:
1. This commit: WITH .cache (baseline)
2. Next commit: WITHOUT .cache (comparison)
3. Analysis: Compare CI run times to quantify cache impact
Documents the empirical testing and performance impact of removing
.cache directory caching from GitHub Actions workflows.

Key findings:
- Removing .cache had minimal to slightly positive impact (-1.2%)
- Cache miss rate was 100% during testing
- Tool-specific caches remain and are more effective
- Recommendation: Proceed with removal

See docs/ci/remove-dot-cache.md for full analysis.
@snomiao snomiao force-pushed the copilot/remove-cache-from-github-action branch from f9c8f81 to 846fb87 Compare October 19, 2025 03:41
@snomiao snomiao marked this pull request as ready for review October 19, 2025 03:46
@snomiao snomiao requested a review from DrJKL October 19, 2025 03:46
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Oct 19, 2025
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking more getting rid of the whole actions/cache step in these cases, not just the .cache directory

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

benchmark updated into pr desc

@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Oct 20, 2025
This commit removes the entire actions/cache steps (not just .cache
directory) from all workflow files to measure the true performance
impact of caching vs no caching.

Changes:
- ci-lint-format.yaml: Removed tool outputs cache
- ci-tests-storybook.yaml: Removed storybook build cache (both jobs)
- ci-tests-unit.yaml: Removed coverage and vitest cache
- api-update-electron-api-types.yaml: Removed tsbuildinfo cache
- api-update-manager-api-types.yaml: Removed tool cache and repo cache
- api-update-registry-api-types.yaml: Removed tool cache and repo cache
- release-draft-create.yaml: Removed tsbuildinfo cache
- release-pypi-dev.yaml: Removed dist and tsbuildinfo cache

Note: pnpm package caching via setup-node still remains active.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Oct 20, 2025
@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Oct 20, 2025
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Oct 20, 2025
@snomiao snomiao requested review from DrJKL and removed request for snomiao October 20, 2025 15:45
@christian-byrne christian-byrne merged commit aa943ac into main Oct 20, 2025
27 checks passed
@christian-byrne christian-byrne deleted the copilot/remove-cache-from-github-action branch October 20, 2025 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: Remove ./.cache caching in github action and then watch how it performs.

4 participants