Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Oct 17, 2025

Overview

Removes .cache directory caching from GitHub Actions workflows to observe performance impact and establish a baseline for future optimization decisions.

Context

As discussed in #5988, the codebase has been evolving with components moving into the /packages directory structure, not just /src. The current cache key hashing may not be capturing all relevant changes that affect the build, making the .cache caching potentially ineffective or misleading.

This PR removes the .cache path from all workflow cache configurations to:

  1. Observe how workflows perform without this caching layer
  2. Establish a performance baseline for comparison
  3. Allow data-driven decisions about re-implementing caching if needed

Changes

Removed .cache from the cache path configuration in 8 workflow files:

  • ci-lint-format.yaml - Kept tool-specific caches (.eslintcache, .prettierCache, .knip-cache, tsconfig.tsbuildinfo)
  • ci-tests-storybook.yaml - Removed from both build and chromatic jobs; kept storybook-static and tsconfig.tsbuildinfo
  • ci-tests-unit.yaml - Kept coverage and .vitest-cache
  • api-update-electron-api-types.yaml - Kept tsconfig.tsbuildinfo
  • api-update-manager-api-types.yaml - Kept tsconfig.tsbuildinfo
  • api-update-registry-api-types.yaml - Kept tsconfig.tsbuildinfo
  • release-draft-create.yaml - Kept tsconfig.tsbuildinfo
  • release-pypi-dev.yaml - Kept dist and tsconfig.tsbuildinfo

Note: Other caching mechanisms remain intact:

  • Tool-specific caches (ESLint, Prettier, Vitest, etc.) are still cached
  • pnpm package caching via cache: 'pnpm' in setup-node actions is unchanged
  • All cache steps still have at least one path entry

Testing

  • ✅ YAML syntax validated for all modified workflow files
  • ✅ All cache steps have non-empty path configurations
  • ✅ No structural changes to workflow logic

Next Steps

After this PR merges, we can monitor workflow performance metrics and decide whether to:

  • Keep the current configuration if performance is acceptable
  • Implement improved caching that accounts for the /packages directory structure
  • Add additional cache paths based on actual build dependencies

Fixes #5988

Original prompt

This section details on the original issue you should resolve

<issue_title>CI: Remove ./.cache caching in github action and then watch how it performs.</issue_title>
<issue_description>Keep in mind, we've started moving some components of the codebase into packages, so not everything that impacts the build is under /src anymore
At that point, how much are we saving with the caching? How would you feel about removing that piece and seeing how it performs, then if it's a problem we can address it but with a baseline for comparison?

Originally posted by @DrJKL in #5949 (comment)

┆Issue is synchronized with this Notion page by Unito
</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #5988


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

┆Issue is synchronized with this Notion page by Unito

@Copilot Copilot AI changed the title [WIP] Remove caching in CI GitHub action for performance evaluation CI: Remove .cache caching from GitHub Actions workflows Oct 17, 2025
Copilot finished work on behalf of snomiao October 17, 2025 01:01
@Copilot Copilot AI requested a review from snomiao October 17, 2025 01:01
Copy link

github-actions bot commented Oct 19, 2025

🎭 Playwright Test Results

⚠️ Tests passed with flaky tests

⏰ Completed at: 10/19/2025, 03:59:25 AM UTC

📈 Summary

  • Total Tests: 494
  • Passed: 456 ✅
  • Failed: 0
  • Flaky: 7 ⚠️
  • Skipped: 31 ⏭️

📊 Test Reports by Browser

  • chromium: View Report • ✅ 447 / ❌ 0 / ⚠️ 7 / ⏭️ 31
  • chromium-2x: View Report • ✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0
  • chromium-0.5x: View Report • ✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0
  • mobile-chrome: View Report • ✅ 6 / ❌ 0 / ⚠️ 0 / ⏭️ 0

🎉 Click on the links above to view detailed test results for each browser configuration.

Copy link

github-actions bot commented Oct 19, 2025

🎨 Storybook Build Status

Build completed successfully!

⏰ Completed at: 10/19/2025, 03:43:41 AM UTC

🔗 Links


🎉 Your Storybook is ready for review!

Copy link

github-actions bot commented Oct 19, 2025

Bundle Size Report

Summary

  • Raw size: 12.5 MB baseline 12.5 MB — ⚪ 0 B
  • Gzip: 2.54 MB baseline 2.54 MB — ⚪ 0 B
  • Brotli: 1.86 MB baseline 1.86 MB — ⚪ 0 B
  • Bundles: 16 current • 16 baseline

Category Glance
App Entry Points ⚪ 0 B (11.7 MB) · Graph Workspace ⚪ 0 B (706 kB) · Panels & Settings ⚪ 0 B (74.8 kB) · UI Components ⚪ 0 B (11.9 kB) · Data & Services ⚪ 0 B (10 kB) · Views & Navigation ⚪ 0 B (7.99 kB) · + 1 more

Per-category breakdown
App Entry Points — 11.7 MB (baseline 11.7 MB) • ⚪ 0 B _Main entry bundles and manifests_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ------------------------ | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/index-CtdCLNdd.js | 9.67 MB | 9.67 MB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/index-DOdupr_x.js | 2.02 MB | 2.02 MB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/index-PESgPnbc.js | 507 B | 507 B | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Graph Workspace — 706 kB (baseline 706 kB) • ⚪ 0 B _Graph editor runtime, canvas, workflow orchestration_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ---------------------------- | ------ | ------ | ------------------ | ------------------ | ------------------ | | assets/GraphView-BtlKFHvX.js | 706 kB | 706 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Views & Navigation — 7.99 kB (baseline 7.99 kB) • ⚪ 0 B _Top-level views, pages, and routed surfaces_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/UserSelectView-DjvAHfN5.js | 7.99 kB | 7.99 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Panels & Settings — 74.8 kB (baseline 74.8 kB) • ⚪ 0 B _Configuration panels, inspectors, and settings screens_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ------------------------------------ | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/AboutPanel-CLKaxNTg.js | 10.1 kB | 10.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/CreditsPanel-CNqxmfsr.js | 21.9 kB | 21.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/ExtensionPanel-HQ5RMX7X.js | 11.9 kB | 11.9 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/KeybindingPanel-B-XQIYLb.js | 15.1 kB | 15.1 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/ServerConfigPanel-Bx6_KxjP.js | 8.04 kB | 8.04 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/UserPanel-BIhpR5Ri.js | 7.76 kB | 7.76 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
UI Components — 11.9 kB (baseline 11.9 kB) • ⚪ 0 B _Reusable component library chunks_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ----------------------------------------------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/ComfyQueueButton-Ca24HJzZ.js | 10.7 kB | 10.7 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/UserAvatar.vue_vue_type_script_setup_true_lang-C9bSkTC5.js | 1.12 kB | 1.12 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Data & Services — 10 kB (baseline 10 kB) • ⚪ 0 B _Stores, services, APIs, and repositories_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | ------------------------------------ | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/keybindingService-DPPz7ASj.js | 7.21 kB | 7.21 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B | | assets/serverConfigStore-CMadDTUK.js | 2.79 kB | 2.79 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |
Utilities & Hooks — 1.07 kB (baseline 1.07 kB) • ⚪ 0 B _Helpers, composables, and utility bundles_ | File | Before | After | Δ Raw | Δ Gzip | Δ Brotli | | --------------------------- | ------- | ------- | ------------------ | ------------------ | ------------------ | | assets/mathUtil-CTARWQ-l.js | 1.07 kB | 1.07 kB | ⚪ 0 B | ⚪ 0 B | ⚪ 0 B |

@snomiao
Copy link
Member

snomiao commented Oct 19, 2025

📊 Performance Analysis: Removal Impact

I've completed comprehensive performance testing to measure the impact of removing .cache directory caching from GitHub Actions workflows.

🎯 Executive Summary

Recommendation:Proceed with removing .cache

Impact: Minimal to slightly positive performance impact (-1.2% total time on average)


📈 Performance Results

Workflow WITH .cache WITHOUT .cache Difference % Change
CI: Tests Unit 173s (2.9m) 174s (2.9m) +1s +0.6%
CI: Lint Format 216s (3.6m) 213s (3.5m) -3s -1.4%
CI: Tests Storybook 103s (1.7m) 99s (1.6m) -4s -3.9%
TOTAL 492s (8.2m) 486s (8.1m) -6s -1.2%

🔍 Key Findings

  1. Cache Miss Rate: 100% 🎯

    • The .cache directory experienced complete cache misses during testing
    • Evidence: Cache not found for input keys: lint-format-cache-Linux-...
    • This explains why removing it had no negative impact
  2. Performance Impact: Slightly Positive

    • No degradation observed
    • Average 1.2% improvement (6 seconds over ~8 minutes)
    • All workflows showed stable or improved performance
  3. Why .cache Was Ineffective 🤔

    • Monorepo evolution: code moved from /src to /packages/
    • Cache keys didn't account for new structure
    • Modern build tools (Vite, Vitest) have efficient built-in caching
    • Tool-specific caches proved more effective

✅ What Caching Remains (and why it's better)

This change preserves all effective caching:

  • Package Manager: pnpm cache via setup-node (saves 2-3 min/run)
  • Tool-Specific Caches:
    • .eslintcache, .prettierCache, .knip-cache
    • .vitest-cache, tsconfig.tsbuildinfo
    • More targeted and effective than generic .cache
  • Build Artifacts: storybook-static, dist, coverage

🧪 Methodology

Two test commits measured real-world performance on GitHub Actions runners:

  1. Baseline (84b4956): Temporarily restored .cache to all 8 workflows
  2. Comparison (2af6f81): Removed .cache again

Test runs:


📖 Full Documentation

Comprehensive analysis available at: docs/ci/remove-dot-cache.md


✨ Benefits of This Change

✅ No performance degradation
✅ Simpler maintenance (fewer cache invalidation issues)
✅ Better cache efficiency (tool-specific > generic)
✅ Clearer intent (each cache serves documented purpose)
✅ Reduced complexity (fewer cache keys to manage)


Conclusion: The data supports proceeding with this PR. Removing .cache simplifies our CI configuration while maintaining (or slightly improving) performance.

Copilot AI and others added 5 commits October 19, 2025 03:41
This commit temporarily restores .cache caching to establish a baseline
for performance comparison. The next commit will remove it again to
measure the performance impact.

Testing approach:
1. This commit: WITH .cache (baseline)
2. Next commit: WITHOUT .cache (comparison)
3. Analysis: Compare CI run times to quantify cache impact
Documents the empirical testing and performance impact of removing
.cache directory caching from GitHub Actions workflows.

Key findings:
- Removing .cache had minimal to slightly positive impact (-1.2%)
- Cache miss rate was 100% during testing
- Tool-specific caches remain and are more effective
- Recommendation: Proceed with removal

See docs/ci/remove-dot-cache.md for full analysis.
@snomiao snomiao force-pushed the copilot/remove-cache-from-github-action branch from f9c8f81 to 846fb87 Compare October 19, 2025 03:41
@snomiao snomiao marked this pull request as ready for review October 19, 2025 03:46
@snomiao snomiao requested a review from DrJKL October 19, 2025 03:46
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Oct 19, 2025
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking more getting rid of the whole actions/cache step in these cases, not just the .cache directory

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: Remove ./.cache caching in github action and then watch how it performs.

3 participants