Skip to content

fix(tests): sandbox CWD for suites that write via $PWD (#526) - #527

Merged
Data-Wise merged 2 commits into
devfrom
feature/fix-test-cwd-isolation
Sep 14, 2026
Merged

Data-Wise merged 2 commits into
devfrom
feature/fix-test-cwd-isolation

Conversation

@Data-Wise

@Data-Wise Data-Wise commented Sep 14, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Fixes test: run-all.sh runs every suite with CWD = repo root, no sandbox #526: tests/run-all.sh cds to the repo root before running every suite, so any test that resolves its write target via $PWD (instead of an explicit path) mutates this repo's own tracked files.
  • Confirmed writers: test-capture.zsh / test-adhd.zsh / e2e-core-commands.zsh (all → .STATUS, via real win/focus calls) and e2e-teach-analyze.zsh (→ tests/fixtures/demo-course/.teach/concepts.json).
  • run_test() gains an opt-in sandbox mode (3rd arg "sandbox"): it cds into a fresh mktemp'd scratch project dir (with a throwaway .STATUS) inside a subshell before invoking the suite, so $PWD-resolved writes land there instead of the real tree. Wired up for the four confirmed suites.
  • e2e-teach-analyze.zsh gets its own fix: it deliberately cds into the real fixture dir to exercise teach analyze, so CWD sandboxing can't help (the fixture path comes from $0, not $PWD). It now backs up concepts.json before the analyze calls and restores it after, with a trap ... EXIT so it restores on early exit too.
  • Added a regression guard at the end of run-all.sh: after the full run, assert git status --short (from the repo root) is empty, failing the run otherwise. This is the systemic gate — it catches any future suite that dirties a tracked file, not just these four.

Why sandboxing isn't the default for every suite

I first tried making the sandbox the default for all 85 registered suites (wrapping run_test unconditionally). That surfaced 10 new failures, all pre-existing latent bugs unrelated to #526: several suites (test-cc-dispatcher.zsh among them) recompute their project root from $0 inside a shell function, where zsh's $0 is the function name, not the script path — they were only ever passing because their $PWD-based fallback happened to match the repo root, which every suite used to inherit. Blanket sandboxing would have traded one class of accidental-CWD bug for another and expanded this PR's blast radius well past #526. Scoped the sandbox to an opt-in instead, applied only to the four confirmed writers.

Test plan

Full suite run in this sandboxed dev environment, comparing before/after (this machine lacks atlas, giving a 3-failure baseline unrelated to this change: test-doctor, e2e-em-dispatcher, test-atlas-contract):

  • Baseline (pre-fix, original run-all.sh): 85 passed / 3 failed / 0 timeout / 0 skipped — reproduced the bug: git status --short afterward showed M .STATUS and M tests/fixtures/demo-course/.teach/concepts.json.
  • Post-fix: 85 passed / 3 failed / 0 timeout / 0 skipped — same 3 pre-existing failures, zero regressions — and git status --short afterward is empty.
  • e2e-teach-analyze.zsh still passes on its own (backup/restore verified via an explicit before/after diff of concepts.json).

Code review round — 5 more CONFIRMED gaps found and fixed

A high-effort review (8 finder angles + 1-vote verify against this diff) surfaced 5 real, empirically-verified gaps in the fix above, all fixed in a second commit:

  1. restore_concepts's trap didn't fire on timeout. trap restore_concepts EXIT alone doesn't run when the suite is killed by run-all.sh's timeout wrapper via SIGTERM — empirically verified (timeout 1 zsh script.zsh with an EXIT trap and sleep 5 never runs it). This suite's own comments say it already runs close to its 30s budget, so a slow/cold-cache run timing out would leave concepts.json corrupted and leak the mktemp backup — defeating the fix under exactly the condition it exists to guard against. Fixed with trap 'restore_concepts; exit 143' TERM INT alongside the existing EXIT trap (idempotent, safe to call from both).
  2. Dirty-tree guard had no pre-run baseline. A bare post-run git status --short misattributes any unrelated pre-existing uncommitted change to test pollution. Added a BASELINE_STATUS snapshot before any suite runs, diffed via comm -13 at the end. Verified both cells: pre-existing dirt alone doesn't fail; pre-existing dirt plus injected new pollution flags only the new line.
  3. Guard reported failure after the Results: line was already printed — a polluting run showed "0 failed" next to a separate, uncounted ❌ block. Moved the check (and its FAIL increment) before the Results: line.
  4. e2e-teach-analyze.zsh was tagged "sandbox" even though CWD-sandboxing is a no-op for it — its write is ${0:A:h}-anchored, never $PWD-derived; the real protection is its own trap. Removed the misleading tag (a future reader could otherwise "clean up" the trap as redundant, silently reopening test: run-all.sh runs every suite with CWD = repo root, no sandbox #526 for this file).
  5. _run_sandboxed's cleanup never runs if run-all.sh itself is Ctrl-C'd mid-suite, leaking a scratch dir in $TMPDIR per interruption. Added CURRENT_SANDBOX tracking + a script-level INT/TERM trap.

Two other suites with the identical unfixed CWD/fixture-write bug were found but are out of scope for this PR — neither is registered in run-all.sh today (tests/integration-teach-macros.zsh, tests/diagnostic-teach-analyze.zsh); noting for a possible follow-up rather than expanding this PR.

Re-verification after the fixes

Full suite: 85 passed / 3 failed / 0 timeout — same 3 pre-existing failures, tree confirmed clean afterward. Positive control re-run: manually injecting a .STATUS change is correctly caught by the guard; reverting it leaves the tree clean again.

Data-Wise and others added 2 commits September 14, 2026 11:19
test-capture.zsh, test-adhd.zsh, and e2e-core-commands.zsh call commands
(`win`, `focus`) that resolve their target through _flow_find_project_root,
which walks up from $PWD rather than taking an explicit path. Since
run-all.sh cd's to the repo root before running every suite, those calls
silently mutated the real .STATUS every time the suite ran.

Add an opt-in sandbox mode to run-all.sh's run_test() (pass "sandbox" as
the 3rd arg): it cd's into a fresh mktemp'd project dir (with a throwaway
.STATUS) inside a subshell before invoking the suite, so $PWD-resolved
writes land there instead. Scoped as an opt-in rather than the default for
every suite: most suites anchor themselves via $0 instead of $PWD and were
never audited against an unfamiliar CWD — test-cc-dispatcher.zsh in
particular recomputes its project root from $0 *inside a function*, where
zsh's $0 is the function name rather than the script path, and was only
ever passing because its $PWD-based fallback happened to match the repo
root every suite used to inherit. Blanket sandboxing would have traded one
class of accidental-CWD bug for another; wire it up per suite instead.

e2e-teach-analyze.zsh has a different bug: it deliberately cd's into
tests/fixtures/demo-course to exercise `teach analyze`, which overwrites
the tracked .teach/concepts.json fixture in place as a side effect. CWD
sandboxing can't fix that (the test resolves the fixture path via $0, not
$PWD), so it gets its own backup/restore around the analyze calls, with a
trap to restore even on early exit.

Also add a regression guard at the end of run-all.sh: after the full run,
assert `git status --short` is empty from the repo root, failing the run
otherwise. This is the systemic gate — it catches any future suite that
mutates a tracked file, not just the four confirmed here.
High-effort code review (8 finder angles + verify) on PR #527's own fix
found 5 CONFIRMED real gaps in the fix itself, all fixed here:

1. restore_concepts's EXIT-only trap in e2e-teach-analyze.zsh doesn't
   fire on a SIGTERM from run-all.sh's `timeout` wrapper (empirically
   verified: `timeout 1 zsh script.zsh` with an EXIT trap and `sleep 5`
   never runs it). A timing-out run — plausible, since this suite's own
   comments say it already runs close to its 30s budget — would leave
   concepts.json corrupted and leak the mktemp backup, defeating the fix
   under exactly the condition it exists to guard against. Added
   `trap 'restore_concepts; exit 143' TERM INT` alongside the existing
   EXIT trap; restore_concepts is idempotent (clears CONCEPTS_BACKUP
   after restoring) so calling it from both handlers is safe.

2. The dirty-tree guard took one post-run `git status --short` snapshot
   with no baseline, so any unrelated pre-existing uncommitted change in
   a developer's own tree was misattributed to test pollution. Added a
   pre-run BASELINE_STATUS snapshot and diff via `comm -13` — verified
   both cells: pre-existing dirt alone doesn't fail; pre-existing dirt
   plus injected new pollution flags only the new line.

3. The guard ran and reported failure *after* the "Results: X passed,
   Y failed..." line was already printed, so a polluting run showed
   "0 failed" next to a separate, uncounted ❌ block. Moved the check
   (and its FAIL increment) before the Results line.

4. e2e-teach-analyze.zsh was tagged `"sandbox"` even though its write
   (concepts.json) is resolved via `${0:A:h}`, never $PWD — CWD-sandboxing
   is a complete no-op for it; its own backup/restore trap is the actual
   protection. Removed the misleading tag; a future reader could otherwise
   credit it and remove the "redundant"-looking trap, silently reopening
   #526 for this file.

5. _run_sandboxed's `rm -rf "$sandbox"` never runs if run-all.sh itself
   is interrupted (Ctrl-C) mid-suite, leaking a scratch dir in $TMPDIR
   per interruption. Added CURRENT_SANDBOX tracking + an INT/TERM trap
   at the script level.

Full suite (local, macOS/BSD): 85 passed / 3 failed / 0 timeout — same
3 pre-existing failures as the documented baseline (test-doctor,
e2e-em-dispatcher, test-atlas-contract), none touching the files this
commit changes. Working tree confirmed clean after the run (the actual
goal of #526) — verified via a positive control: manually injecting a
.STATUS change is correctly caught by the guard, and reverting it
leaves the tree clean again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Data-Wise
Data-Wise merged commit afab47b into dev Sep 14, 2026
3 checks passed
@Data-Wise
Data-Wise deleted the feature/fix-test-cwd-isolation branch September 14, 2026 17:54
Data-Wise added a commit that referenced this pull request Sep 14, 2026
Repo-tracked copy of the session report (PR #525/#527 merged,
issues #524-527 resolved, #487 parked) — the .remember/ original
is gitignored and machine-local.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant