Skip to content

Conversation

Aditi-1400
Copy link
Contributor

@Aditi-1400 Aditi-1400 commented Jun 23, 2025

Adds an option (NODE_COMPILE_CACHE_RELATIVE_PATH) for the built-in compile cache to encode the hashes with relative file paths. On enabling the option,
the source directory along with cache directory can be bundled and moved, and the cache continues to work.

When enabled, paths encoded in hash are relative to compile cache directory.

Fixes: #58755
Refs: #52696

Thanks @joyeecheung for all the help :)

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jun 23, 2025
Copy link

codecov bot commented Jun 23, 2025

Codecov Report

❌ Patch coverage is 86.07595% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.89%. Comparing base (9bcc5a8) to head (b879f6b).
⚠️ Report is 279 commits behind head on main.

Files with missing lines Patch % Lines
src/compile_cache.cc 81.48% 1 Missing and 4 partials ⚠️
src/path.cc 73.68% 2 Missing and 3 partials ⚠️
src/env.cc 87.50% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #58797      +/-   ##
==========================================
+ Coverage   89.37%   89.89%   +0.52%     
==========================================
  Files         654      667      +13     
  Lines      192554   196670    +4116     
  Branches    37434    38617    +1183     
==========================================
+ Hits       172086   176794    +4708     
+ Misses      12995    12331     -664     
- Partials     7473     7545      +72     
Files with missing lines Coverage Δ
lib/internal/modules/helpers.js 98.29% <100.00%> (+0.92%) ⬆️
src/compile_cache.h 100.00% <ø> (ø)
src/env.h 98.14% <ø> (ø)
src/node_modules.cc 77.06% <100.00%> (-0.25%) ⬇️
src/env.cc 80.83% <87.50%> (-0.05%) ⬇️
src/compile_cache.cc 80.78% <81.48%> (-0.01%) ⬇️
src/path.cc 72.81% <73.68%> (+3.76%) ⬆️

... and 195 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Member

@joyeecheung joyeecheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add some integration to module.enableCompileCache() as well? I think having it in options bag when the first argument is an object might work well enough. The path can be options.path in that case.

@joyeecheung joyeecheung mentioned this pull request Jul 3, 2025
13 tasks
@mcollina mcollina added the semver-minor PRs that contain new features and should be released in the next minor version. label Jul 4, 2025
@Aditi-1400 Aditi-1400 force-pushed the compile-cache branch 2 times, most recently from 5abcfc9 to de78d1a Compare August 1, 2025 13:15
Copy link
Member

@jasnell jasnell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file:// URL handling in this is incorrect. Either support for file:// URLs here at all should be dropped or they should be correctly handled.

jasnell

This comment was marked as duplicate.

@Aditi-1400 Aditi-1400 force-pushed the compile-cache branch 4 times, most recently from ddd3bcd to d03bf27 Compare August 11, 2025 16:30
@jasnell
Copy link
Member

jasnell commented Aug 23, 2025

Great work @Aditi-1400 !

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@joyeecheung
Copy link
Member

joyeecheung commented Aug 25, 2025

19:41:32 not ok 190 parallel/test-compile-cache-portable-esm
19:41:32   ---
19:41:32   duration_ms: 227.01200
19:41:32   severity: fail
19:41:32   exitcode: 1
19:41:32   stack: |-
19:41:32     node:fs:1325
19:41:32       const result = binding.mkdir(
19:41:32                              ^
19:41:32     
19:41:32     Error: ENOENT: no such file or directory, mkdir 'c:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.192\$tmp\dir with $unusual"chars?'åß∂ƒ©∆¬…\work'
19:41:32         at Object.mkdirSync (node:fs:1325:26)
19:41:32         at Object.<anonymous> (c:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-compile-cache-portable-esm.js:16:4)
19:41:32         at Module._compile (node:internal/modules/cjs/loader:1737:14)
19:41:32         at Object..js (node:internal/modules/cjs/loader:1870:10)
19:41:32         at Module.load (node:internal/modules/cjs/loader:1469:32)
19:41:32         at Module._load (node:internal/modules/cjs/loader:1289:12)
19:41:32         at TracingChannel.traceSync (node:diagnostics_channel:322:14)
19:41:32         at wrapModuleLoad (node:internal/modules/cjs/loader:238:24)
19:41:32         at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5)
19:41:32         at node:internal/main/run_main_module:33:47 {
19:41:32       errno: -4058,
19:41:32       code: 'ENOENT',
19:41:32       syscall: 'mkdir',
19:41:32       path: `c:\\workspace\\node-test-binary-windows-js-suites\\node\\test\\.tmp.192\\$tmp\\dir with $unusual"chars?'åß∂ƒ©∆¬…\\work`
19:41:32     }
19:41:32     
19:41:32     Node.js v25.0.0-pre
19:41:32   ...

The test is failing on Windows - it looks more like a pre-existing problem of mkdirSync, or a problem with the test.

Adds an option (NODE_COMPILE_CACHE_PORTABLE) for
the built-in compile cache to encode the hashes with
relative file paths. On enabling the option,
the source directory along with cache directory can be
bundled and moved, and the cache continues to work.

When enabled, paths encoded in hash are relative to
compile cache directory.
@Aditi-1400 Aditi-1400 added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 28, 2025
@github-actions github-actions bot added request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. and removed request-ci Add this label to start a Jenkins CI on a PR. labels Aug 28, 2025
Copy link
Contributor

Failed to start CI
   ⚠  Commits were pushed since the last approving review:
   ⚠  - src: add an option to make compile cache portable
   ✘  Refusing to run CI on potentially unsafe PR
https://github.com/nodejs/node/actions/runs/17295846847

@Aditi-1400 Aditi-1400 added request-ci Add this label to start a Jenkins CI on a PR. and removed request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. request-ci Add this label to start a Jenkins CI on a PR. labels Aug 28, 2025
@Aditi-1400 Aditi-1400 requested a review from joyeecheung August 28, 2025 15:37
@joyeecheung joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Sep 8, 2025
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 8, 2025
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@joyeecheung joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Sep 11, 2025
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 11, 2025
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@Aditi-1400 Aditi-1400 added the commit-queue Add this label to land a pull request using GitHub Actions. label Sep 12, 2025
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Sep 12, 2025
@nodejs-github-bot nodejs-github-bot merged commit 94422e8 into nodejs:main Sep 12, 2025
73 checks passed
@nodejs-github-bot
Copy link
Collaborator

Landed in 94422e8

targos pushed a commit that referenced this pull request Sep 12, 2025
Adds an option (NODE_COMPILE_CACHE_PORTABLE) for
the built-in compile cache to encode the hashes with
relative file paths. On enabling the option,
the source directory along with cache directory can be
bundled and moved, and the cache continues to work.

When enabled, paths encoded in hash are relative to
compile cache directory.

PR-URL: #58797
Fixes: #58755
Refs: #52696
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. semver-minor PRs that contain new features and should be released in the next minor version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support relative compile cache
5 participants