OS watch limits due to filtering occurring after watch registration

[jonatanolofsson]

Issue description

Part of this issue is also raised at the watchfiles repository. Here I will also include a (temporary) workaround for the interested reader.

To watch for changes, sphinx-autobuild uses watchfiles and I've encountered an issue with how notifications are registered in very large directories. Namely: watchfiles listens for notifications on all files, then when notified decides whether to filter out the file or not. I.e., as opposed to only watching a filtered set of files for changes.

In my project, there is the equivalent of a .cache-directory with a very large number of files. Thus, when listening for changes in the repository, watchfiles tries to register watches on all files, leading to the following OS error:

OSError: OS file watch limit reached.
about ["/path/to/..."]
(Error { kind: MaxFilesWatch, paths: [...] })

Using the watch_filter is to no avail, as all the watches are still being registered, and the filtering only occurs after changes have been reported.

sphinx-autobuild automatically registers the entire sphinx project source directory for watching, meaning that if this includes the cache-directory, sphinx-autobuild will not work due to the above error. Ignoring the cache-directory

Workaround

In sphinx-autobuild, there is a setup procedure which collects which directories should be watched. Again, this always include the project source directory. These are passed to a _create_app function, and this is where it is possible to intercept this list and modify it through monkey-patching:

#!/usr/bin/env python3
import sys
from pathlib import Path

import sphinx_autobuild.__main__ as autobuild_main_module


# --- Backup the original _create_app ---
original_create_app = autobuild_main_module._create_app


# --- Monkey-patched _create_app ---
def patched_create_app(watch_dirs, ignore_handler, builder, out_dir, url_host):
    new_watch_dirs = []

    for d in watch_dirs:
        d_path = Path(d)
        if not d_path.is_dir():
            new_watch_dirs.extend(d)
            continue

        # Add all direct subdirectories that do not start with '.'
        subdirs = [str(p) for p in d_path.iterdir() if p.is_dir() and not p.name.startswith('.')]
        new_watch_dirs.extend(subdirs)

    print(f"[patched_create_app] Modified watch_dirs:\n{new_watch_dirs}")
    return original_create_app(new_watch_dirs, ignore_handler, builder, out_dir, url_host)


# --- Apply the monkey patch ---
autobuild_main_module._create_app = patched_create_app


# --- Run sphinx-autobuild ---
if __name__ == "__main__":
    # Forward command-line arguments
    sys.exit(autobuild_main_module.main())