Repository navigation
fix(watcher): file watcher honours the scanner's pruning; gitignored churn no longer rebuilds the index (2.129.1) - #177
Merged
Conversation
…churn no longer rebuilds the index (2.129.1) services/watcher.py kept its own thirteen-entry skip list and tracked any .json/.md/.yaml under the project, ignoring the root .gitignore and linked worktrees that scanner.iter_files already prunes. On a project whose daemons rewrite four state files under a gitignored logs/ every few seconds, the change counter passed IndexStalenessAgent's rebuild threshold every minute: "Index auto-rebuilt" reached a count of 75,968, four agent threads burned ~5 CPU hours in 17 hours on an idle session, and each refresh burst left the MCP server's process heap with pinned 16 MB segments it never released (28 GB commit, <1 GB working set, one heap, ~1,770 segments). Three such servers exhausted the box's commit charge and the C3 Hub answered with MemoryError from werkzeug, which presented as "the hub fails to start". _ChangeHandler now filters with scanner.make_dir_pruner (shared SKIP_DIRS, root .gitignore directory entries and single-segment globs) and is_nested_checkout (cached per directory), before the sub-project excluder and the extension allowlist; editing the root .gitignore reloads the pruner. A change counts only when the file could be in the index. tests/test_watcher_pruning.py covers gitignored dirs, globs, the reload, worktrees, scanner-only skip names, the excluder and the rootless legacy constructor. Claude-Session: https://claude.ai/code/session_01Uw6AAAWX5CVG3QgThjC2UU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
services/watcher.pykept its own thirteen-entry skip list and tracked any.json/.md/.yamlunder the project, ignoring the root.gitignoreand linked worktrees thatscanner.iter_filesalready prunes.On a project whose daemons rewrite four state files under a gitignored
logs/every few seconds, the change counter passedIndexStalenessAgent's rebuild threshold (15) every minute:MemoryErrorfrom inside werkzeug, which presented as "the hub fails to start"The fix
_ChangeHandlernow filters with the scanner's own pruning, in this order:scanner.make_dir_pruner(sharedSKIP_DIRSsuperset, root.gitignoredirectory entries and single-segment globs),is_nested_checkout(cached per directory, one stat per directory rather than per event), the sub-project excluder, then the extension allowlist. A change counts only when the file could be in the index. Editing the root.gitignorereloads the pruner. The rootless_ChangeHandler()constructor keeps working with the shared skip list.Tests
tests/test_watcher_pruning.py(9 tests): gitignored dirs, gitignore globs, the reload on.gitignoreedit, linked worktrees, the project root never treated as nested, scanner-only skip names, the extension allowlist, the sub-project excluder, and the legacy constructor.Full suite on the final tree: 4058 passed, 7 skipped, 2 failed. The 2 are
test_codex_mcp_transport.py::test_stdio_host_session, the known transport flake on this box (chromadb import lock at server start); it fails the same way onmainin a clean worktree.Not in this PR
The per-refresh heap growth itself (~0.7 GB of pinned segments per
CodeIndex.refreshburst) still exists for legitimate churn. With this fix a refresh needs 15 real source changes instead of 15 log rewrites, so it stops being continuous, but what refresh allocates deserves its own look.https://claude.ai/code/session_01Uw6AAAWX5CVG3QgThjC2UU