[EtanHey]

Why

BrainLayer's own guards sent every SIGKILL (2026-10-28, M4 Max, 1.5.78).

• supervisor/guard-events.jsonl and supervisor/calendar/guard-events.jsonl record guard-timeout, reason heartbeat-timeout:
  • 04:50:05Z: daemon pid 40731, last_heartbeat_age_s 31.2
  • 06:49:01Z: daemon pid 84637, 35.8 s
  • 06:49:02Z: calendar runner pid 82172, 36.0 s
• Right after each emit, run_guard calls os.kill(parent, SIGKILL). The SIGTERM that stopped backup-daily is the same guard's reap_groups (SIGTERM, then SIGKILL, to the registered pgids).
• No agent, jetsam or reaper sent these signals. Agent JSONLs (Claude and Codex) only show kills of non-BrainLayer PIDs.
• Trigger: both parents are launchd spawn type = background (5) with low-priority I/O. Under load 13–16 and about 10 GB of swap they went more than 30 s without a heartbeat, longer than the old guard_timeout=30 allows.
• A dead parent is still caught at once: pipe EOF makes os.read return b"", the loop breaks, and the groups are reaped. The timeout only matters for a parent that is alive but silent, so 120 s does not delay cleanup after a crash.

Change

• supervisor_guard.configured_guard_timeout() is the one resolver. Default 120 s. Override with BRAINLAYER_SUPERVISOR_GUARD_TIMEOUT_SECONDS, from the BrainLayer env file (which wins, as for BRAINLAYER_BACKUP_TIMEOUT_SECONDS) or from the environment. The value must be finite and 10 ≤ v ≤ 900. Anything else raises ValueError naming the variable; there is no silent fallback. That includes an env-file assignment using $(...) or backticks. The shared parser skips such a line as absent, so the resolver detects it for this knob and raises; config.py is unchanged for every other knob (round-1 fix).
• Supervisor(guard_timeout=None) and run_calendar(guard_timeout=None) resolve through it. An explicit argument still wins, so the existing sub-second test fixtures are unchanged.
• run_guard is untouched. The resolver sits at the top of the module, away from the region the parallel host-stall PR (W2, fix/guard-host-stall) edits.
• docs/operations/supervisor-jobs.md updated.
• fix(calendar): keep custody I/O off the heartbeat loop #1519's per-operation budget (operation_timeout 120 s) and the overrun-kill path are unchanged.

Deadline audit (every value derived from guard_timeout)

Site	Before (30)	After (120)	Verdict
service_supervisor.py loop sleep min(0.1, guard_timeout/4)	0.1 s	0.1 s	Cadence unchanged. ≤0.1 s for any allowed value (the floor of 10 gives 2.5 → 0.1)
maintenance_calendar.py _execute_step poll min(0.1, guard_timeout/4)	0.1 s	0.1 s	unchanged
maintenance_calendar.py pulse(..., min(0.1, guard_timeout/4)) heartbeat	0.1 s	0.1 s	unchanged
_admit_calendar full-invocation admission deadline max(step.timeout, 7500) + 60 + guard_timeout + 2*shutdown + 30	7650 s	7740 s	+90 s, intended. This term covers a previous runner's guard expiring, so it should grow with the guard
launch_guard(startup_timeout=30) readiness	30 s	30 s	independent of the heartbeat timeout
run_guard quiesced (Q) deadline shutdown_timeout + 2	17 s	17 s	independent
reap_groups(timeout=shutdown_timeout)	15 s	15 s	independent
Overrun-kill: overdue op → heartbeats withheld → guard expires	120 + 30 = 150 s	120 + 120 = 240 s	Path unchanged as the brief asks. The stuck-worker kill now lands one guard timeout later, which is the trade-off that stops false kills

Nothing else reads guard_timeout (grep -rn guard_timeout src/brainlayer). wal_checkpoint's _guard_timeout_seconds is a separate checkpoint-lock knob and is not affected.

Tests (written first, seen red)

• tests/test_supervisor_guard.py
  • test_guard_timeout_defaults_to_120_seconds
  • test_guard_timeout_env_override_and_env_file_override
  • test_invalid_guard_timeout_raises_never_falls_back[environ|env-file × 0, -5, nan, inf, -inf, 9.99, 900.5, 901, abc, ""] (20 cases)
  • test_default_guard_survives_60s_silent_parent_and_kills_past_120s: in-process run_guard on an injected clock and an honest scripted select, with no real sleep. A 60 s silence is survived; the SIGKILL lands exactly 120 s after the last heartbeat.
  • test_eof_from_dead_parent_reaps_at_once_regardless_of_timeout: EOF at t=5 s reaps immediately, with no kill.
  • test_unsupported_env_file_substitution_raises_never_falls_back[$(bad), bad, "$(echo 60)", 6echo 0 × with/without process override 240] (8 cases; red on 4b0f36b, green on the fix) (round 1)
  • test_substitution_on_other_knobs_keeps_the_shared_parser_contract (round 1)
• tests/test_service_supervisor.py::test_supervisor_guard_timeout_defaults_to_120_and_reads_override
• tests/test_maintenance_calendar.py::test_calendar_guard_timeout_defaults_to_120_and_reads_override

Local results:

• Round 1: the 41 resolver/default/timing tests and all of tests/test_supervisor_guard.py (51) passed; the scoped pre-push passed.
• New tests: 26 passed.
• Scoped regression over test_supervisor_guard, test_service_supervisor, test_maintenance_calendar{,_completion,_leases} and test_supervisor_jobs: 193 passed in 158 s.
• ruff check and ruff format --check clean.
• Pre-push ran BRAINLAYER_PREPUSH_SCOPE=changed-only and passed.

Not run: the full suite, Swift tests, and live-DB tests (host under memory pressure, per the brief). There is no live deploy or probe; this PR is not deployed.

Loop record

Loop-record: implementer=claude/claude-opus-5-5/medium reviewer=codex/sol rounds=1/0 defects=1 role=worker

• Round 1 (Codex, cross-vendor, on head 4b0f36b9): CHANGES REQUESTED, one P2. Defect: an env-file value using $(...) or backticks was skipped by the shared parser and silently fell back to 120 s or the process value. Fixed in 5bb07b2d, with 8 regression cases plus an other-knob contract test.
• Merge order (lead): fix(supervisor): stop guard false kills under host starvation (30s→120s, configurable) #1537 lands first. fix(supervisor): re-arm guards on host stall instead of killing #1536 (fix/guard-host-stall) rebases onto it afterwards; the reviewer notes a textual conflict at the module constants, and both blocks should be kept.
• CodeRabbit and Bugbot are unavailable on this PR (review-limit and usage-limit exhausted); there is no bot review.
• Not merged. The lead owns the merge.

— brainlayerClaude (worker) · claude-code/claude-opus-5-5

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes supervisor/calendar liveness enforcement defaults and startup validation for a critical timeout knob; stuck-but-alive parents now survive longer before SIGKILL, and bad config fails hard instead of falling back.

Overview
Fixes false guard-timeout SIGKILLs when background launchd parents stall 31–36s under load by raising the shared heartbeat budget from 30s to 120s for the daemon and maintenance calendar.

Adds supervisor_guard.configured_guard_timeout() as the single resolver: default 120s, override via BRAINLAYER_SUPERVISOR_GUARD_TIMEOUT_SECONDS (BrainLayer env file wins over process env), validated finite 10–900; invalid or substitution-heavy env lines raise at startup instead of silently falling back. Supervisor and run_calendar now default guard_timeout=None and resolve through this helper; explicit constructor/argument values still override (tests keep sub-second fixtures).

Docs in supervisor-jobs.md describe the new knob and note that overdue executor work still suppresses heartbeats, so a wedged worker is killed one guard timeout after the last heartbeat (longer than before). run_guard behavior is unchanged; pipe EOF still reaps dead parents immediately.

Tests cover defaults, overrides, bounds, env-file substitution errors, and scripted timing (60s silence survives; kill at 120s after last heartbeat; EOF reaps at once).

^{Reviewed by Cursor Bugbot for commit 5bb07b2. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Raise supervisor guard timeout default from 30s to 120s and make it configurable

• Adds configured_guard_timeout in supervisor_guard.py: default 120s, override via BRAINLAYER_SUPERVISOR_GUARD_TIMEOUT_SECONDS (env file takes precedence over process env), valid range 10–900s inclusive. Invalid, non-finite, or out-of-range values raise ValueError instead of falling back to the default.
• Supervisor.__init__ in service_supervisor.py and run_calendar in maintenance_calendar.py now use the configured value when no explicit guard_timeout is passed; explicit arguments still win.
• Environment-file assignments using command substitution for the timeout variable are rejected with ValueError.
• Risk: invalid configuration now fails startup (constructor and calendar admission) instead of silently using 30s; guard no longer kills a silent parent before 120s, so a wedged process lingers longer by default.

^{Macroscope summarized 5bb07b2.}

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 30 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 29fe8b96-6ae9-4a3d-aa31-965eb16ce506

Commits

Reviewing files that changed from the base of the PR and between afcef22 and 5bb07b2.

Files selected for processing: 7

• docs/operations/supervisor-jobs.md
• src/brainlayer/maintenance_calendar.py
• src/brainlayer/service_supervisor.py
• src/brainlayer/supervisor_guard.py
• tests/test_maintenance_calendar.py
• tests/test_service_supervisor.py
• tests/test_supervisor_guard.py

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: dc17a0a0-c11e-4e42-8524-ac8b59bed38d)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: d4d80a52-10b3-4612-8d27-89391b1e3608)

[github-actions]

BrainLayer ratchet

Every Value below was measured by this run. A row this machine cannot measure says n/a — <reason> instead of a number; baselines in Notes name their own machine, method and date and were not measured here.

Hosted success gives no verdict on mapped bytes, search p50/p95, idle CPU, or fallback replay debt.

Row	Status	Value (measured by this run)	Method	Notes
commit provenance	🟢 GREEN	measured 5bb07b2d56a6 == PR head · checkout 5bb07b2d56a6	commit graph + live PR head · in-process · runner	Which commit this whole table is about. On a pull_request event the checkout is GitHub's synthetic merge ref, whose sha is not on the PR — #759's table printed 13fa724278bf while that PR's head was 4632f979 — so this row names the PR-head parent instead, the sha a reviewer can actually see. The comparison sha is read live from repos/{owner}/{repo}/pulls/{n} when the table is collected, not taken from the event payload, because the payload cannot know the run has been overtaken. Residual window, stated rather than papered over: a push landing between that read and the comment being posted is not caught here — the run for that push refreshes the table.
baseline attestation	🟢 GREEN	baseline f421d1a7c5e6 matches the main attestation (run 40663077231 · main 1253f606dc59 · 2026-10-28T09:17:53Z)	main attestation artifact via Actions API · in-process · runner	What every comparison is measured AGAINST, and who says so. The baseline fields of tests/fixtures/sprint_gate/corpus.json (queries, latency_baseline_ms, thresholds) are compared to the ratchet-attestation artifact of the latest successful push or (no-input) workflow_dispatch run of ratchet-attest.yml on main, fetched through the Actions API — a PR run cannot write to another run's artifacts. A field that differs is RED unless that main run measured the new value. The calibrated socket collector can license p50/p95; every absent measured path stays locked, so missing collection never passes as permission for a hand edit. Boundary: the comparator is this PR's checkout of ci_ratchet_table.py, diff-reviewable, not tamper-proof.
provenance	🟢 GREEN	stamped 5bb07b2d56a6 == HEAD, tree clean	wheel stamp · in-process · runner	Sha half of #749 keg-mode provenance: a keg built from this wheel can answer __build_sha__. The helper-age and served-process predicates need a running BrainBar and are measured only by scripts/sprint_gate.py on an installed Mac. The sha here is the checkout's — the explicit PR head in ratchet.yml — because that is what publish.yml stamps at release time; the PR-head sha this table describes is the one in commit provenance above.
fallback replay debt	⚪ n/a	n/a — no fallback queue on this machine: the pending memories live in ~/Gits/*/docs.local/decisions, and docs.local/ is gitignored, so a runner checkout has no copy of them to count	docs.local walk · machine with the fallback queue	local-only; not enforced by hosted CI. intended_brain_store: true with no chunk_id means a memory reached disk and never reached the DB, so it answers no brain_search. Budget: 0. Any pending or unparseable file is a finding, never a band -- 122 of these sat from 2026-06-28 to 2026-09-05 because nothing counted them where a reader would look. Measured by walking the tree, so it is only ever measured on a machine that HAS the tree.
pre-push: delete-only push runs no suite	🟢 GREEN	0 suite starts · exit 0	real checkout hook · delete-only stdin · sentinel suite runner	Bug SHA: 44ecbf5ba6f1ada3b965797559eee90c94b45595 (RED replay, 2026-10-06). Fix: #1084; measured at 5bb07b2d56a61f6b6bfa024e9ce513f0ba9af85c. Ceiling: 0 suite starts; hook must exit 0 and announce the delete-only skip.
mapped bytes	⚪ n/a	n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would	socket · installed Mac	local-only; not enforced by hosted CI. Baseline 26.2 GB — installed Mac, socket, 2026-09-03, after R2 drained 15,070 → 0. Up from 16.8 GB because the drain left more vectors mapped under the same cap: the change is the drain, not a leak. Not measured by this run.
search p50/p95	⚪ n/a	n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would	socket · installed Mac	local-only; not enforced by hosted CI. Margin p50: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin p95: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Calibrated on MacBook-Pro.local at 2026-09-01T08:42:22Z under active_sprint_load (tests/fixtures/sprint_gate/corpus.json). Not measured by this run.
idle CPU	⚪ n/a	n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would	ps sampling · installed Mac	local-only; not enforced by hosted CI. Ceiling: average CPU < 30% over a 60 s window (resource_budget in scripts/sprint_gate.py), ratified and kept as a hard budget. Margin daemon: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin helper: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin watcher: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Needs the BrainBar daemon, helper and watcher actually running. Not measured by this run.
signature_valid	⚪ n/a	n/a — the macOS signature-parity job is trigger-gated and did not run on this PR: it touches no release or signing path (pyproject.toml, scripts/release-*, scripts/brainlayer-version-check.sh, publish.yml, ratchet.yml) and carries no ratchet:signatures label — a GitHub macOS runner bills at ~10× Linux minutes and rebuilds the keg venv from source	codesign · installed keg	scripts/release-verify-signatures.sh <keg> codesign-verifies every *.so/*.dylib under libexec/venv. The macOS parity job installs the published tap formula (etanhey/layers/brainlayer), so this row measures the release path — formula, published sdist and Homebrew's relocation — and not this PR's tree. Release-time baseline for the same keg on a different machine: 442 valid / 0 invalid — installed Mac (M4 Max), brew --prefix brainlayer 1.5.11, 2026-09-03.
BrainBar renders no enrichment status	🟢 GREEN	19 actual-view captures; active pending-store/replay retained	macOS DEBUG binary · AppKit actual views · Vision OCR + icon RGBA comparison · private synthetic state	Missing render capability/evidence is RED. Historical data is retained; this is source-build proof, not installed-app proof.
Watcher heartbeat freshness and recovery	⚪ n/a	n/a — macOS native probe trigger-gated: no affected path or ratchet:heartbeat label	private Python producer -> compiled Swift consumer	Preserves genuine silence; independent agent stores cannot revive the watcher. Not a load or installed-UI measurement.
maintenance quiesce: services stay down under a reviving watchdog	⚪ n/a	n/a — macOS replay trigger-gated: no affected path or ratchet:quiesce label	real launchd · GitHub macOS runner · real maintenance + fleet-watchdog	Fix: #1076 (bc6d690f4331387485678d0247e19e14e6ef8d97). Ceiling: 0 revivals; supervisor held, both resumed, no hold marker.
no cloud model call reachable anywhere	🟢 GREEN	508 source files; 444 installed imports; 3 native tests; wheel 1960e386fd17; 0 candidate attempts	immutable installed wheel (default + dev) + AST source + actual native BrainBar	Synthetic private fixtures; SDK absence gated on merged R10b. Exact-head candidate proof, not deployment.

🟢 GREEN measured, within budget · 🔴 RED measured, out of budget — a finding to clear before merge · ⚪ n/a not measurable on this machine, never guessed.

No RED rows.

Measured on Linux/x86_64 · measured 5bb07b2d56a6 · PR head 5bb07b2d56a6 · checkout 5bb07b2d56a6 · run · updated 2026-10-28 09:19:43 UTC