Skip to content

fix(supervisor): re-arm guards on host stall instead of killing - #1536

Merged
EtanHey merged 6 commits into
mainfrom
fix/guard-host-stall
Oct 28, 2026
Merged

EtanHey merged 6 commits into
mainfrom
fix/guard-host-stall

Conversation

@EtanHey

@EtanHey EtanHey commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Problem and scope

This is a backstop for a host-wide stall that freezes the guard too. It would NOT have stopped the 2026-10-28 04:50 kill: that guard's observed wait overshoot was only 1.2 seconds, below the 5-second stall threshold. The separate 120-second timeout PR (fix/guard-timeout-120) is the primary fix for the 10-28 incident. Do not ship this PR alone as the incident fix.

The shared daemon/calendar guard detects starvation when its own monotonic select wait exceeds the requested wait by more than 5 seconds. This avoids wall-clock adjustment false positives and coordination across guards. It does not detect sleep on platforms where monotonic time pauses.

Each re-arm grants a full heartbeat timeout window, shortened only by the remaining cumulative grace allowance. Grace is capped at BOTH three re-arms AND 300 seconds of granted windows per incident, whichever is reached first. At timeout 120 seconds, the grants are 120, 120, then 60 seconds; at timeout 900 seconds, the single grant is 300 seconds. Host time during which the guard cannot run is not counted as a granted window. Only a real H resets both caps; registration traffic does not.

EOF is consumed before considering stall grace: a dead parent's registered process groups are reaped without a re-arm or timeout kill. A responsive guard still kills a silent parent on its normal timeout, including operation-overrun heartbeat suppression. Shutdown (Q) deadlines never receive grace. Heartbeat defaults and operation budgets are unchanged by this PR.

guard-stall-rearm uses the existing bounded asynchronous diagnostics sink, recording observed wait overshoot, re-arm count, cumulative granted seconds, parent PID/birth and guard PID. Last-heartbeat age continues to describe a real heartbeat.

Round 1 response

Claude cross-vendor review of ab3e525e94dc551117923fa1584f456154f18d2d: VERDICT OK, with follow-ups in review comment. This is the one authorized follow-up round in the same PR:

  • Added the 300-second cumulative grace cap, plus tests at 120- and 900-second timeouts and a heartbeat-reset test.
  • Added stall-plus-EOF coverage proving group reaping with no kill or re-arm; adjusted EOF handling to meet that contract.
  • Corrected scope and grace wording: each re-arm grants a full timeout window except where the 300-second cap truncates it. The prior automated "~15 extra seconds" claim was incorrect.
  • The 120-second change is now PR #1537, OPEN and unmerged at the follow-up push. No rebase was needed. If it merges first, preserve its GUARD_TIMEOUT_* block alongside these STALL_*/MAX_STALL_* constants when rebasing.

Loop-record: implementer=codex/gpt-6.1-sol/high reviewer=claude/unknown rounds=1/0 defects=2 role=worker

The round-1 verdict covers the prior head. The lead owns verification of this follow-up head and merge. Do not merge from this worker.

Validation

  • Initial TDD: 8 expected failures before production edits, followed by 13 passing new cases.
  • Round-1 TDD: 4 new expected failures and 13 passes before production edits. After the fix, scoped guard, diagnostics, supervisor-overrun subprocess and calendar-overrun tests: 41 passed. Pytest reported its existing unset asyncio fixture-scope deprecation.
  • ruff check, ruff format --check, and git diff --check passed.
  • New round-1 test names: test_cumulative_stall_grace_is_capped_at_300_seconds (120/900), test_real_heartbeat_resets_cumulative_stall_grace, test_stall_coinciding_with_eof_reaps_without_rearm.
  • Existing tests still cover 30/60/90-second stalls, responsive hangs, three-rearm exhaustion and heartbeat reset, registration traffic, shutdown, diagnostic failure, and operation suppression.
  • No live DB, launchd/service, installation or release changes. No full Python or Swift suite ran locally. Deployed behavior remains for the lead's batch verification.
  • Historical pre-push attempt with system pytest failed because pytest-timeout was missing; the original push passed using the normal uv dev runner. No gate was bypassed.
  • Round-1 changed-only push passed its full gate: 35 guard cases, 3 MCP registration cases, 40 isolated eval/hook cases, 5 Bun tests, and the FTS5 shell regression. Head: 40d5a576bf714ec149778d71945607d20fe89c09.

size:M because the expanded regression coverage proves both grace bounds, reset and EOF behavior at the 120/900-second timeout limits.

CodeRabbit local review: SKIPPED — rate-limit exhaustion (exit 1), including the follow-up attempt; no paid retry. The original PR review was also rate-limited. Manual diff inspection and scoped tests supplement the lead-routed cross-vendor review, not replace it. Neither Bugbot nor Greptile was requested by this worker.

Evidence and handoff

Preserve the earlier superseded crash-sweep gate failure: CHANGES_RESULT: cancelled, empty SWEEPS_REQUIRED, exit 1 before running a sweep. No manual retry was requested to erase that result. Current-head CI state is reported in the follow-up handoff; old-head green does not certify the new head.

Worktree disposition: RETAINED by the W2 implementer for the open PR and lead review; T3 owns eventual retirement.

— brainlayerCodex (worker) · codex/gpt-6.1-sol

🤖 Generated with Claude Code.

Note

Re-arm supervisor_guard.run_guard deadlines on host stall instead of killing

  • Adds stall re-arming in supervisor_guard.py: when a select wait overshoots by more than STALL_OVERSHOOT_S (5s), the guard moves its deadline instead of killing the parent, with at most 3 re-arms and an absolute grace end 300s (MAX_STALL_GRACE_S) after the missed deadline.
  • A real H heartbeat resets the re-arm count and grace end; EOF, quiesced shutdown, and silent parents still get the normal timeout kill. Failed stall diagnostics do not disable enforcement.
  • Extends the full-calendar admission budget in maintenance_calendar.py by MAX_STALL_GRACE_S so admission can outwait a prior guard that used its stall grace.
  • Adds a deterministic run_guard simulator in test_supervisor_guard_stalls.py and a calendar admission test in test_calendar_recovery_priority.py.
  • Behavioral Change: parents stalled by host scheduling now get up to 300s of extra grace before the heartbeat-timeout kill; check supervisor_guard.run_guard stall handling and the re-arm constants in supervisor_guard.

Macroscope summarized 07e9843.


Note

Medium Risk
Changes supervisor parent-kill timing and extends full-calendar admission waits by up to 300s; bounded by tests but affects critical maintenance custody paths.

Overview
When the supervisor guard’s select wait overshoots the requested timeout by more than 5 seconds, it now treats that as host starvation rather than a silent parent: it re-arms the heartbeat deadline (up to 3 times and within a 300s absolute grace window from the first miss), emits guard-stall-rearm diagnostics, and only then falls back to the existing guard-timeout / parent kill. Real H heartbeats reset stall counters; EOF is handled before stall logic so dead parents still exit cleanly; quiesced (Q) guards never re-arm.

Full calendar admission budgets in _admit_calendar now add MAX_STALL_GRACE_S so a scheduled run can keep waiting for custody while a prior owner’s guard finishes its stall grace, avoiding premature calendar-admission-timeout.

Adds deterministic test_supervisor_guard_stalls.py coverage for re-arm caps, grace accounting, heartbeat/registration/shutdown edge cases, plus a calendar priority test that admission succeeds when the prior guard “expires late.”

Reviewed by Cursor Bugbot for commit 07e9843. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of temporary scheduling delays so active supervisor checks can recover without triggering a premature timeout.
    • Updated calendar admission timing to account for the additional recovery window, helping scheduled plans proceed when a prior check experiences a delay.

Co-Authored-By: brainlayerCodex running gpt-6.1-sol <noreply@anthropic.com>
@EtanHey EtanHey added the size:S Tight-loop PR size: 51-150 hand-written lines added label Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 47 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 4a5e3cf5-03fa-4d2b-8b60-1f621af3285f
Commits

Reviewing files that changed from the base of the PR and between bebe99b and 07e9843.

Files selected for processing: 3
  • src/brainlayer/maintenance_calendar.py
  • src/brainlayer/supervisor_guard.py
  • tests/test_supervisor_guard_stalls.py

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 32c9971c-d406-466d-aae0-9bf638c01a89
Commits

Reviewing files that changed from the base of the PR and between f6a044b and bebe99b.

Files selected for processing: 4
  • src/brainlayer/maintenance_calendar.py
  • src/brainlayer/supervisor_guard.py
  • tests/test_calendar_recovery_priority.py
  • tests/test_supervisor_guard_stalls.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Recent review details
Context from checks skipped due to timeout: 15
  • GitHub Check: crash sweeps
  • GitHub Check: swift (macos-15)
  • GitHub Check: BrainBar renders no enrichment status
  • GitHub Check: signature parity trigger
  • GitHub Check: test (3.11)
  • GitHub Check: test (3.12)
  • GitHub Check: installed wheel imports (python 3.13)
  • GitHub Check: test (3.13)
  • GitHub Check: global model retirement (installed + native)
  • GitHub Check: Macroscope - Correctness Check
  • GitHub Check: Analyze (swift)
  • GitHub Check: Analyze (actions)
  • GitHub Check: Analyze (python)
  • GitHub Check: Analyze (c-cpp)
  • GitHub Check: Analyze (javascript-typescript)
Additional context used
Code guidelines: 1
AGENTS.md — auto-discovered
Path-based instructions: 1
Source excerpt: **The guard's one contract:** in `tests/`, reach an embedding model through `brainlayer.embeddings` or a module attribute — never `from sentence_transformers import X`.

CodeRabbit inference engine (AGENTS.md)

Files:

  • tests/test_calendar_recovery_priority.py
  • tests/test_supervisor_guard_stalls.py
Additional comments: 4
src/brainlayer/supervisor_guard.py: 1

26-28: LGTM!

Also applies to: 285-286, 293-294, 296-324, 333-333, 366-367

tests/test_supervisor_guard_stalls.py: 1

1-134: LGTM!

src/brainlayer/maintenance_calendar.py: 1

32-32: LGTM!

Also applies to: 648-650, 655-655

tests/test_calendar_recovery_priority.py: 1

214-257: LGTM!

Walkthrough

The supervisor guard now rearms after qualifying wait overshoots, subject to re-arm and cumulative-grace caps. A real heartbeat resets both caps. Calendar admission includes the maximum stall grace in its deadline.

Changes: Supervisor Guard Stall Grace

Layer / File(s) Summary
Guard re-arm behavior
src/brainlayer/supervisor_guard.py, tests/test_supervisor_guard_stalls.py
The guard rearms after wait overshoots greater than five seconds, up to three times and 300 seconds of cumulative grace. A real heartbeat resets both limits. Tests cover re-arm limits, timeout behavior, EOF, quiescence, and diagnostic failures.
Calendar admission deadline
src/brainlayer/maintenance_calendar.py, tests/test_calendar_recovery_priority.py
Calendar admission adds the maximum stall grace to its deadline. A regression test checks that admission succeeds after the recovery, exit, and grace allowances.

Priority: Pending latest changes

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant SupervisorGuard
  participant HeartbeatPipe
  participant ParentProcess
  participant DiagnosticSink
  SupervisorGuard->>HeartbeatPipe: wait for heartbeat input
  HeartbeatPipe-->>SupervisorGuard: input after scheduling stall
  SupervisorGuard->>DiagnosticSink: emit guard-stall-rearm
  SupervisorGuard->>HeartbeatPipe: wait for re-armed timeout
  HeartbeatPipe-->>SupervisorGuard: real heartbeat
  SupervisorGuard->>SupervisorGuard: reset stall-rearm counters
  SupervisorGuard->>ParentProcess: kill after timeout when heartbeat is absent
Loading

Possibly related issues

  • Issue 1445: This PR extends the calendar admission deadline for guard stall grace. It does not implement the issue’s indexing and finalization budget changes.
  • Issue 1422: This PR extends calendar custody timing with bounded stall grace. It does not implement the backup receipt deadline or startup sequencing.

Possibly related PRs

  • EtanHey/brainlayer#1464: This PR modifies the heartbeat guard and its timeout behavior established by that change.
  • EtanHey/brainlayer#1525: This PR changes guard behavior for the heartbeat-starvation condition and adds tests for it.
Merge Risk: Low · previous review bebe9

Repeated prolonged stalls can cause a full-calendar run to time out while the previous guard retains custody.

Pre-merge checks: Passed 3 · Failed 2

Failed Failed checks: 2 warnings

Check name Status Explanation Resolution
Description check Warning The description covers the problem, scope, re-arm and grace rules, EOF and quiescence handling, validation, and handoff notes, which are the substance this check needs. It does not follow the reposito… Add the template's required sections. Use the '## Change and evidence' heading, and describe the change and the exact-head checks run. Add the '## Crash sweep / feature batch' section. State whether the crash-sweep label is needed, and repo…
Docstring Coverage Warning Docstring coverage is 11.11% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 3 passed
Check name Status Explanation
Title check Passed The title names the main change: guards now re-arm on host stalls instead of killing the parent. It follows the conventional-commit style and matches the supervisor_guard changes, the stall constants,…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: guards now re-arm on host stalls instead of killing the parent. It follows the conventional-commit style and matches the supervisor_guard changes, the stall constants, and the re-arm tests in the changeset. It is 63 characters, within a reasonable length.

Full details: Description check

Explanation

The description covers the problem, scope, re-arm and grace rules, EOF and quiescence handling, validation, and handoff notes, which are the substance this check needs. It does not follow the repository template. It has no 'Change and evidence' heading. It has no 'Crash sweep / feature batch' section, and the crash-sweep gate instructions are not addressed beyond a historical note. The Loop-record line is present but its values are for the author workflow. The description is largely complete but misses required template sections.

Resolution

Add the template's required sections. Use the '## Change and evidence' heading, and describe the change and the exact-head checks run. Add the '## Crash sweep / feature batch' section. State whether the crash-sweep label is needed, and report the exact-head CI status for the batch commit SHA with the verify-release gate result. Keep the Loop-record line in the template's format.

Full details: Docstring Coverage

Explanation

Docstring coverage is 11.11% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 18 functions across 4 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit waits while seconds sprawl,
The guard grants grace, but counts it all.
A heartbeat hops; the caps reset,
The calendar waits past the sunset.
Then quiet timeouts close the gate,
And plans resume without delay.

Comment @coderabbitai help to get the list of available commands.

@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 8568ee0b-02e7-4246-a1b6-f2e93f0f8faf)

@EtanHey

EtanHey commented Oct 28, 2026

Copy link
Copy Markdown
Owner Author

Cross-vendor review — #1536 @ ab3e525

VERDICT: OK
Read-only review: no pytest/swift/builds run. CI at review time: NOT green. The superseded run's crash sweeps FAILED (CHANGES_RESULT cancelled); the current run's test 3.11/3.12/3.13, swift and crash sweeps are still in progress.

Answers to the brief

  1. Signal soundness: sound. overshoot = now - wait_started - remaining (supervisor_guard.py:261-265) is > 5 s only when the guard woke more than 5 s past its own deadline. An early wake on data gives a negative overshoot. EOF and data arrive as readable, so they never take the continue at :281-282.
  2. Hung parent / dead parent: a hung parent on a responsive host (overshoot ≤ 5 s) is killed exactly as before (:295-300). A dead parent gives pipe EOF, which is readable, falls through to os.read → break (:320-322), so it is never masked. A hung parent on a stalling host survives at most 3 extra timeout windows.
  3. Cap reset: correct. Only H while not quiesced resets it (:326-329). +/-pgid and lease L/U do not. Tested at test_supervisor_guard_stalls.py:78.
  4. Shutdown/quiesce: unchanged. The not quiesced gate (:268) means a Q deadline (:330-332) gets no grace, and the EOF/finally/reap path is untouched. Tested at tests:91.
  5. Diagnostics: non-blocking. Module emit swallows all exceptions (supervisor_diagnostics.py:115-121), and DiagnosticSink.emit is put_nowait and drops when the queue is full (:43-50). Tested at tests:98.
  6. Tests: clock and select are injected, with no real sleeps. All five required cases are covered (tests:46, :57, :64, :103, :72), plus registration, quiesce and diag-failure cases.
  7. 120 s interaction: intents are compatible. See F2 and F3.

Findings

  • F1 MEDIUM (scope, not a code bug). On its own, this PR would not have prevented the 04:50:05Z kill. The deadline is set only at H, so at the final wake overshoot = last_heartbeat_age − timeout. The incident ages were 31.2 / 35.8 / 36.0 s, so the overshoots were 1.2 / 5.8 / 6.0 s. That is under the 5 s threshold at 04:50, and only marginally over it at 06:49. In this incident the parent starved, but the tiny resident guard mostly did not, and a guard-self signal cannot see that. The 120 s timeout PR is the actual incident fix; fix(supervisor): re-arm guards on host stall instead of killing #1536 is defense in depth for whole-host freezes (guard starved too). Ask: say this in the PR body so nobody ships fix(supervisor): re-arm guards on host stall instead of killing #1536 alone as "the 10-28 fix".
  • F2 LOW (merge mechanics). The 120 s worktree (t3-f3f08f19, not yet pushed) adds its GUARD_TIMEOUT_* constants and configured_guard_timeout() in the same spot, right after the supervisor_diagnostics import (supervisor_guard.py:24-27). Whoever lands second gets a trivial textual conflict. Keep both blocks.
  • F3 LOW (bound scales with timeout). Grace is MAX_STALL_REARMS * timeout (:279). At 120 s that is up to 360 s of extra windows, so a hung parent on a stalling host can live about 480 s plus overshoots. At the 120 s PR's 900 s upper bound it is about 3600 s. That meets the brief's "≤3 re-arms or ≤300 s", but only via the re-arm clause. Optional: cap cumulative grace too, e.g. min(timeout, 300 - used).
  • F4 LOW (observability). When a stall coincides with a readable H, the guard emits guard-stall-rearm and then immediately resets the cap (:268-282 then :327). That is a true starvation signal, but readers should not treat it as "grace consumed". Acceptable as is.
  • F5 LOW (test hygiene). The fixture patches the global os.write, os.close, os.read, os.kill and time.monotonic modules (tests:31-37), not guard-local seams, for the duration of run_guard. That is safe, because it fails toward no real signals, but it is fragile if pytest-timeout or capture threads touch them. Prefer injected seams later.
  • F6 LOW (test gap). No case covers a stall that coincides with EOF, i.e. a dead parent during a stall: simulate_guard([(60, b"")]) should exit with no kill and one rearm row. The code is correct by reading (:281 → :320-322), but brief Q2 ("dead parent never masked") deserves a pin.
  • F7 INFO. The Macroscope summary in the PR body says "~15 extra seconds of grace". That is wrong: each re-arm grants a full timeout window (:270).

Required before merge

  • CI green on ab3e525 (or its successor), with the crash-sweeps gate passing rather than cancelled.
  • PR body: add the F1 scope note. Recommended: land the 120 s PR first and rebase this one onto it (F2), and add the F6 test.

— Claude reviewer (cross-vendor)

Co-Authored-By: brainlayerCodex running gpt-6.1-sol <noreply@anthropic.com>
@EtanHey EtanHey added size:M Tight-loop PR size: 151-400 hand-written lines added and removed size:S Tight-loop PR size: 51-150 hand-written lines added labels Oct 28, 2026
@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: d71c3022-080d-47bb-bb9e-9e43497db129)

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

BrainLayer ratchet

Every Value below was measured by this run. A row this machine cannot measure says n/a — <reason> instead of a number; baselines in Notes name their own machine, method and date and were not measured here.

Hosted success gives no verdict on mapped bytes, search p50/p95, idle CPU, or fallback replay debt.

Row Status Value (measured by this run) Method Notes
commit provenance 🟢 GREEN measured 07e984353db1 == PR head · checkout 07e984353db1 commit graph + live PR head · in-process · runner Which commit this whole table is about. On a pull_request event the checkout is GitHub's synthetic merge ref, whose sha is not on the PR — #759's table printed 13fa724278bf while that PR's head was 4632f979 — so this row names the PR-head parent instead, the sha a reviewer can actually see. The comparison sha is read live from repos/{owner}/{repo}/pulls/{n} when the table is collected, not taken from the event payload, because the payload cannot know the run has been overtaken. Residual window, stated rather than papered over: a push landing between that read and the comment being posted is not caught here — the run for that push refreshes the table.
baseline attestation 🟢 GREEN baseline f421d1a7c5e6 matches the main attestation (run 40693933060 · main f6a044b45897 · 2026-10-28T13:10:29Z) main attestation artifact via Actions API · in-process · runner What every comparison is measured AGAINST, and who says so. The baseline fields of tests/fixtures/sprint_gate/corpus.json (queries, latency_baseline_ms, thresholds) are compared to the ratchet-attestation artifact of the latest successful push or (no-input) workflow_dispatch run of ratchet-attest.yml on main, fetched through the Actions API — a PR run cannot write to another run's artifacts. A field that differs is RED unless that main run measured the new value. The calibrated socket collector can license p50/p95; every absent measured path stays locked, so missing collection never passes as permission for a hand edit. Boundary: the comparator is this PR's checkout of ci_ratchet_table.py, diff-reviewable, not tamper-proof.
provenance 🟢 GREEN stamped 07e984353db1 == HEAD, tree clean wheel stamp · in-process · runner Sha half of #749 keg-mode provenance: a keg built from this wheel can answer __build_sha__. The helper-age and served-process predicates need a running BrainBar and are measured only by scripts/sprint_gate.py on an installed Mac. The sha here is the checkout's — the explicit PR head in ratchet.yml — because that is what publish.yml stamps at release time; the PR-head sha this table describes is the one in commit provenance above.
fallback replay debt ⚪ n/a n/a — no fallback queue on this machine: the pending memories live in ~/Gits/*/docs.local/decisions, and docs.local/ is gitignored, so a runner checkout has no copy of them to count docs.local walk · machine with the fallback queue local-only; not enforced by hosted CI. intended_brain_store: true with no chunk_id means a memory reached disk and never reached the DB, so it answers no brain_search. Budget: 0. Any pending or unparseable file is a finding, never a band -- 122 of these sat from 2026-06-28 to 2026-09-05 because nothing counted them where a reader would look. Measured by walking the tree, so it is only ever measured on a machine that HAS the tree.
pre-push: delete-only push runs no suite 🟢 GREEN 0 suite starts · exit 0 real checkout hook · delete-only stdin · sentinel suite runner Bug SHA: 44ecbf5ba6f1ada3b965797559eee90c94b45595 (RED replay, 2026-10-06). Fix: #1084; measured at 07e984353db1eca6b008a7ad0c454f369f73e50b. Ceiling: 0 suite starts; hook must exit 0 and announce the delete-only skip.
mapped bytes ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would socket · installed Mac local-only; not enforced by hosted CI. Baseline 26.2 GB — installed Mac, socket, 2026-09-03, after R2 drained 15,070 → 0. Up from 16.8 GB because the drain left more vectors mapped under the same cap: the change is the drain, not a leak. Not measured by this run.
search p50/p95 ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would socket · installed Mac local-only; not enforced by hosted CI. Margin p50: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin p95: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Calibrated on MacBook-Pro.local at 2026-09-01T08:42:22Z under active_sprint_load (tests/fixtures/sprint_gate/corpus.json). Not measured by this run.
idle CPU ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would ps sampling · installed Mac local-only; not enforced by hosted CI. Ceiling: average CPU < 30% over a 60 s window (resource_budget in scripts/sprint_gate.py), ratified and kept as a hard budget. Margin daemon: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin helper: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin watcher: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Needs the BrainBar daemon, helper and watcher actually running. Not measured by this run.
signature_valid ⚪ n/a n/a — the macOS signature-parity job is trigger-gated and did not run on this PR: it touches no release or signing path (pyproject.toml, scripts/release-*, scripts/brainlayer-version-check.sh, publish.yml, ratchet.yml) and carries no ratchet:signatures label — a GitHub macOS runner bills at ~10× Linux minutes and rebuilds the keg venv from source codesign · installed keg scripts/release-verify-signatures.sh <keg> codesign-verifies every *.so/*.dylib under libexec/venv. The macOS parity job installs the published tap formula (etanhey/layers/brainlayer), so this row measures the release path — formula, published sdist and Homebrew's relocation — and not this PR's tree. Release-time baseline for the same keg on a different machine: 442 valid / 0 invalid — installed Mac (M4 Max), brew --prefix brainlayer 1.5.11, 2026-09-03.
BrainBar renders no enrichment status 🟢 GREEN 19 actual-view captures; active pending-store/replay retained macOS DEBUG binary · AppKit actual views · Vision OCR + icon RGBA comparison · private synthetic state Missing render capability/evidence is RED. Historical data is retained; this is source-build proof, not installed-app proof.
Watcher heartbeat freshness and recovery ⚪ n/a n/a — macOS native probe trigger-gated: no affected path or ratchet:heartbeat label private Python producer -> compiled Swift consumer Preserves genuine silence; independent agent stores cannot revive the watcher. Not a load or installed-UI measurement.
maintenance quiesce: services stay down under a reviving watchdog ⚪ n/a n/a — macOS replay trigger-gated: no affected path or ratchet:quiesce label real launchd · GitHub macOS runner · real maintenance + fleet-watchdog Fix: #1076 (bc6d690f4331387485678d0247e19e14e6ef8d97). Ceiling: 0 revivals; supervisor held, both resumed, no hold marker.
no cloud model call reachable anywhere 🟢 GREEN 508 source files; 444 installed imports; 3 native tests; wheel b83d4ad6db97; 0 candidate attempts immutable installed wheel (default + dev) + AST source + actual native BrainBar Synthetic private fixtures; SDK absence gated on merged R10b. Exact-head candidate proof, not deployment.

🟢 GREEN measured, within budget · 🔴 RED measured, out of budget — a finding to clear before merge · ⚪ n/a not measurable on this machine, never guessed.

No RED rows.

Measured on Linux/x86_64 · measured 07e984353db1 · PR head 07e984353db1 · checkout 07e984353db1 · run · updated 2026-10-28 13:53:28 UTC

EtanHey and others added 3 commits October 28, 2026 15:10
# Conflicts:
#	src/brainlayer/supervisor_guard.py
A prior full calendar's guard can now outlive guard_timeout by up to
MAX_STALL_GRACE_S of host-stall re-arms (this PR). The full-calendar
admission deadline counted guard_timeout only, so the next invocation
could give up with calendar-admission-timeout while the prior guard was
still legitimately alive. Add MAX_STALL_GRACE_S to the bound.

Found by the #1537 x #1536 combined-timing re-check (R-1537-codex.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 2071f1b0-c348-4bf1-9b89-6beafdcddb29)

R-1536-delta-sol P2: MAX_STALL_GRACE_S capped the granted windows, not
the elapsed extension. Each re-arm happens after an overshoot > 5 s that
was never counted, so three 36 s overshoots at timeout=120 stretched the
guard to 528 s against the calendar admission's 420 s allowance.

The first stall of an incident now fixes stall_grace_end = missed deadline
+ MAX_STALL_GRACE_S; every re-arm deadline is min(now + timeout, end) and
no re-arm happens at or past it. A real heartbeat clears the incident.
grace_used_s now reports how far the new deadline sits past the missed one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 9bc6627f-ccc9-448c-a376-4a0061a9c8ae)

@EtanHey

EtanHey commented Oct 28, 2026

Copy link
Copy Markdown
Owner Author

Review record: Codex Sol r1 (CHANGES REQUESTED: per-grant stall accounting, fixed in 21312f4 + 07e9843) → r2 (CHANGES REQUESTED: readable-past-end and unbounded final wake latency). The lead check accepts r2 as a pre-existing, bounded run_guard ordering plus the existing finite admission-failure policy; it is tracked as a follow-up row. Full text: R-1536-r2-sol.md (lead check appended). Canon #9 cap reached; merging on green.

@EtanHey
EtanHey merged commit b714c11 into main Oct 28, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M Tight-loop PR size: 151-400 hand-written lines added

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant