🐛 Bug: Not able to watch files with huge node_modules anymore after bumping chokidar to v4

[nescalante]

Bug Report Checklist

• I have read and agree to Mocha's Code of Conduct and Contributing Guidelines
• I have searched for related issues and issues with the faq label, but none matched my issue.
• I have 'smoke tested' the code to be tested by running it outside the real test suite to get a better sense of whether the problem is in the code under test, my usage of Mocha, or Mocha itself.
• I want to provide a PR to resolve this

Expected

mocha --watch should work with big node_modules codebases

Actual

mocha --watch stopped to work with big node_modules codebases

Minimal, Reproducible Example

After merging #5256 I am not able to watch files anymore and instead I am receiving this error:

Error: EMFILE: too many open files, watch
    at FSWatcher._handle.onchange (node:internal/fs/watchers:207:21)
Emitted 'error' event on FSWatcher instance at:
    at FSWatcher._handleError (/Users/nescalante/dev/exchange-ui/node_modules/mocha/node_modules/chokidar/index.js:540:18)
    at NodeFsHandler._boundHandleError (/Users/nescalante/dev/exchange-ui/node_modules/mocha/node_modules/chokidar/handler.js:302:49)
    at /Users/nescalante/dev/exchange-ui/node_modules/mocha/node_modules/chokidar/handler.js:149:9
    at foreach (/Users/nescalante/dev/exchange-ui/node_modules/mocha/node_modules/chokidar/handler.js:81:9)
    at fsWatchBroadcast (/Users/nescalante/dev/exchange-ui/node_modules/mocha/node_modules/chokidar/handler.js:148:5)
    at FSWatcher.<anonymous> (/Users/nescalante/dev/exchange-ui/node_modules/mocha/node_modules/chokidar/handler.js:197:17)
    at FSWatcher.emit (node:events:517:28)
    at FSWatcher._handle.onchange (node:internal/fs/watchers:213:12) {
  errno: -24,
  syscall: 'watch',
  code: 'EMFILE',
  filename: null
}

Node.js v18.20.8

This is because now the watcher is being initialized this way:

const watcher = chokidar.watch('.', {
  ignoreInitial: true
});

and that is causing chokidar to do a watch in the whole directory

Versions

Edit by @mark-wiemer:
Mocha ^11.2.2
Node 18.20.8

Additional Info

No response

[JoshuaKGoldberg]

Oh, that's fun. Do you have a reproduction project you can share? We tested Mocha with Chokdar v4 on a few non-trivial open source repositories but clearly nothing at the scale you're working with.

I'll also note that Chokidar has had EMFILE: too many open files in the past: e.g. paulmillr/chokidar#1369 -> paulmillr/chokidar@710ee7c. It's possible this is an "issue" upstream from Mocha. Hard to say without a reproduction.

[nescalante]

Oh, that's fun. Do you have a reproduction project you can share? We tested Mocha with Chokdar v4 on a few non-trivial open source repositories but clearly nothing at the scale you're working with.

I'll also note that Chokidar has had EMFILE: too many open files in the past: e.g. paulmillr/chokidar#1369 -> paulmillr/chokidar@710ee7c. It's possible this is an "issue" upstream from Mocha. Hard to say without a reproduction.

will create one, it basically explodes when your node_modules is huge 😱

[nescalante]

@JoshuaKGoldberg this repo has a bunch of dependencies making it fail: https://github.com/nescalante/mocha-EMFILE
That one effectively fails in my Apple M3 Max 96GB RAM with default configurations

on the other hand, this other branch with the mocha version downgraded works:
https://github.com/nescalante/mocha-EMFILE/tree/working-version

[nescalante]

Also, the older version starts immediatly and the latest version takes some time until it fails (debugging it ended up being that it keeps loading all files in the node_modules folder)

[Arnesfield]

Might also be related to #5355.

Currently, the watch mode does a glob.sync() on the watchIgnore paths, which is probably expensive and would take a while especially if node_modules is huge.

mocha/lib/cli/watch-run.js

Lines 147 to 151 in 5d617f3

	regenerate() {
	const watchIgnoreSet = new Set();
	for (const pattern of this.watchIgnore) {
	glob.sync(pattern, { dot: true }).forEach(filePath => watchIgnoreSet.add(filePath));
	}

There's now a pull request (#5379) that refactors the watch mode to work around globs (since chokidar v4 removed support for them) which should hopefully resolve this issue as well.

[JoshuaKGoldberg]

Interesting, trying npm run watch on https://github.com/nescalante/mocha-EMFILE locally on my machine after the (very long) npm i doesn't have any issues:

 $ npm run watch

> mocha-emfile@1.0.0 watch
> mocha --watch



  this is my test
good
    ✔ and looks good


  1 passing (1ms)

ℹ [mocha] waiting for changes...

But opening a separate terminal does give the error:

$ npm run watch

> mocha-emfile@1.0.0 watch
> mocha --watch

node:internal/fs/watchers:214
      const error = new UVException({
                    ^

Error: EMFILE: too many open files, watch
    at FSWatcher._handle.onchange (node:internal/fs/watchers:214:21)
Emitted 'error' event on FSWatcher instance at:
    at FSWatcher._handleError (/Users/josh/repos/mocha/node_modules/chokidar/index.js:539:18)
    ...

Fun. Accepting PRs!