fix(tests): sandbox CWD for suites that write via $PWD (#526) - #527
Merged
Merged
Conversation
test-capture.zsh, test-adhd.zsh, and e2e-core-commands.zsh call commands (`win`, `focus`) that resolve their target through _flow_find_project_root, which walks up from $PWD rather than taking an explicit path. Since run-all.sh cd's to the repo root before running every suite, those calls silently mutated the real .STATUS every time the suite ran. Add an opt-in sandbox mode to run-all.sh's run_test() (pass "sandbox" as the 3rd arg): it cd's into a fresh mktemp'd project dir (with a throwaway .STATUS) inside a subshell before invoking the suite, so $PWD-resolved writes land there instead. Scoped as an opt-in rather than the default for every suite: most suites anchor themselves via $0 instead of $PWD and were never audited against an unfamiliar CWD — test-cc-dispatcher.zsh in particular recomputes its project root from $0 *inside a function*, where zsh's $0 is the function name rather than the script path, and was only ever passing because its $PWD-based fallback happened to match the repo root every suite used to inherit. Blanket sandboxing would have traded one class of accidental-CWD bug for another; wire it up per suite instead. e2e-teach-analyze.zsh has a different bug: it deliberately cd's into tests/fixtures/demo-course to exercise `teach analyze`, which overwrites the tracked .teach/concepts.json fixture in place as a side effect. CWD sandboxing can't fix that (the test resolves the fixture path via $0, not $PWD), so it gets its own backup/restore around the analyze calls, with a trap to restore even on early exit. Also add a regression guard at the end of run-all.sh: after the full run, assert `git status --short` is empty from the repo root, failing the run otherwise. This is the systemic gate — it catches any future suite that mutates a tracked file, not just the four confirmed here.
High-effort code review (8 finder angles + verify) on PR #527's own fix found 5 CONFIRMED real gaps in the fix itself, all fixed here: 1. restore_concepts's EXIT-only trap in e2e-teach-analyze.zsh doesn't fire on a SIGTERM from run-all.sh's `timeout` wrapper (empirically verified: `timeout 1 zsh script.zsh` with an EXIT trap and `sleep 5` never runs it). A timing-out run — plausible, since this suite's own comments say it already runs close to its 30s budget — would leave concepts.json corrupted and leak the mktemp backup, defeating the fix under exactly the condition it exists to guard against. Added `trap 'restore_concepts; exit 143' TERM INT` alongside the existing EXIT trap; restore_concepts is idempotent (clears CONCEPTS_BACKUP after restoring) so calling it from both handlers is safe. 2. The dirty-tree guard took one post-run `git status --short` snapshot with no baseline, so any unrelated pre-existing uncommitted change in a developer's own tree was misattributed to test pollution. Added a pre-run BASELINE_STATUS snapshot and diff via `comm -13` — verified both cells: pre-existing dirt alone doesn't fail; pre-existing dirt plus injected new pollution flags only the new line. 3. The guard ran and reported failure *after* the "Results: X passed, Y failed..." line was already printed, so a polluting run showed "0 failed" next to a separate, uncounted ❌ block. Moved the check (and its FAIL increment) before the Results line. 4. e2e-teach-analyze.zsh was tagged `"sandbox"` even though its write (concepts.json) is resolved via `${0:A:h}`, never $PWD — CWD-sandboxing is a complete no-op for it; its own backup/restore trap is the actual protection. Removed the misleading tag; a future reader could otherwise credit it and remove the "redundant"-looking trap, silently reopening #526 for this file. 5. _run_sandboxed's `rm -rf "$sandbox"` never runs if run-all.sh itself is interrupted (Ctrl-C) mid-suite, leaking a scratch dir in $TMPDIR per interruption. Added CURRENT_SANDBOX tracking + an INT/TERM trap at the script level. Full suite (local, macOS/BSD): 85 passed / 3 failed / 0 timeout — same 3 pre-existing failures as the documented baseline (test-doctor, e2e-em-dispatcher, test-atlas-contract), none touching the files this commit changes. Working tree confirmed clean after the run (the actual goal of #526) — verified via a positive control: manually injecting a .STATUS change is correctly caught by the guard, and reverting it leaves the tree clean again. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/run-all.shcds to the repo root before running every suite, so any test that resolves its write target via$PWD(instead of an explicit path) mutates this repo's own tracked files.test-capture.zsh/test-adhd.zsh/e2e-core-commands.zsh(all →.STATUS, via realwin/focuscalls) ande2e-teach-analyze.zsh(→tests/fixtures/demo-course/.teach/concepts.json).run_test()gains an opt-in sandbox mode (3rd arg"sandbox"): itcds into a freshmktemp'd scratch project dir (with a throwaway.STATUS) inside a subshell before invoking the suite, so$PWD-resolved writes land there instead of the real tree. Wired up for the four confirmed suites.e2e-teach-analyze.zshgets its own fix: it deliberatelycds into the real fixture dir to exerciseteach analyze, so CWD sandboxing can't help (the fixture path comes from$0, not$PWD). It now backs upconcepts.jsonbefore the analyze calls and restores it after, with atrap ... EXITso it restores on early exit too.run-all.sh: after the full run, assertgit status --short(from the repo root) is empty, failing the run otherwise. This is the systemic gate — it catches any future suite that dirties a tracked file, not just these four.Why sandboxing isn't the default for every suite
I first tried making the sandbox the default for all 85 registered suites (wrapping
run_testunconditionally). That surfaced 10 new failures, all pre-existing latent bugs unrelated to #526: several suites (test-cc-dispatcher.zshamong them) recompute their project root from$0inside a shell function, where zsh's$0is the function name, not the script path — they were only ever passing because their$PWD-based fallback happened to match the repo root, which every suite used to inherit. Blanket sandboxing would have traded one class of accidental-CWD bug for another and expanded this PR's blast radius well past #526. Scoped the sandbox to an opt-in instead, applied only to the four confirmed writers.Test plan
Full suite run in this sandboxed dev environment, comparing before/after (this machine lacks
atlas, giving a 3-failure baseline unrelated to this change:test-doctor,e2e-em-dispatcher,test-atlas-contract):run-all.sh): 85 passed / 3 failed / 0 timeout / 0 skipped — reproduced the bug:git status --shortafterward showedM .STATUSandM tests/fixtures/demo-course/.teach/concepts.json.git status --shortafterward is empty.e2e-teach-analyze.zshstill passes on its own (backup/restore verified via an explicit before/after diff ofconcepts.json).Code review round — 5 more CONFIRMED gaps found and fixed
A high-effort review (8 finder angles + 1-vote verify against this diff) surfaced 5 real, empirically-verified gaps in the fix above, all fixed in a second commit:
restore_concepts's trap didn't fire on timeout.trap restore_concepts EXITalone doesn't run when the suite is killed byrun-all.sh'stimeoutwrapper via SIGTERM — empirically verified (timeout 1 zsh script.zshwith an EXIT trap andsleep 5never runs it). This suite's own comments say it already runs close to its 30s budget, so a slow/cold-cache run timing out would leaveconcepts.jsoncorrupted and leak the mktemp backup — defeating the fix under exactly the condition it exists to guard against. Fixed withtrap 'restore_concepts; exit 143' TERM INTalongside the existing EXIT trap (idempotent, safe to call from both).git status --shortmisattributes any unrelated pre-existing uncommitted change to test pollution. Added aBASELINE_STATUSsnapshot before any suite runs, diffed viacomm -13at the end. Verified both cells: pre-existing dirt alone doesn't fail; pre-existing dirt plus injected new pollution flags only the new line.Results:line was already printed — a polluting run showed "0 failed" next to a separate, uncounted ❌ block. Moved the check (and itsFAILincrement) before theResults:line.e2e-teach-analyze.zshwas tagged"sandbox"even though CWD-sandboxing is a no-op for it — its write is${0:A:h}-anchored, never$PWD-derived; the real protection is its own trap. Removed the misleading tag (a future reader could otherwise "clean up" the trap as redundant, silently reopening test: run-all.sh runs every suite with CWD = repo root, no sandbox #526 for this file)._run_sandboxed's cleanup never runs ifrun-all.shitself is Ctrl-C'd mid-suite, leaking a scratch dir in$TMPDIRper interruption. AddedCURRENT_SANDBOXtracking + a script-levelINT/TERMtrap.Two other suites with the identical unfixed CWD/fixture-write bug were found but are out of scope for this PR — neither is registered in
run-all.shtoday (tests/integration-teach-macros.zsh,tests/diagnostic-teach-analyze.zsh); noting for a possible follow-up rather than expanding this PR.Re-verification after the fixes
Full suite: 85 passed / 3 failed / 0 timeout — same 3 pre-existing failures, tree confirmed clean afterward. Positive control re-run: manually injecting a
.STATUSchange is correctly caught by the guard; reverting it leaves the tree clean again.