Skip to content

fix(brainbar): tolerate torn ps rows in the activation scan - #1538

Merged
EtanHey merged 1 commit into
mainfrom
fix/activation-scan-tolerant
Oct 28, 2026
Merged

EtanHey merged 1 commit into
mainfrom
fix/activation-scan-tolerant

Conversation

@EtanHey

@EtanHey EtanHey commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Etan sees a latched critical alert, brainbar-activation => "BrainBar activation monitor refused: unreadable process table". It fired at 2026-10-27 10:46 and 2026-10-28 06:49 local, both host-stall moments (load 13-16, swap ~10 GB).

Root cause

  • brew_processes() runs /bin/ps -axo pid=,ppid=,comm=,args= and raised ValueError("unreadable process table") on ANY row that did not split into 4 fields with a decimal pid. One transient torn row was enough, for example a process exiting or exec-ing mid-scan with empty args. (A live scan had 871 rows and 0 malformed ones, so the error is transient.)
  • _defer_for_brew converted only OSError/SubprocessError into the counted ProcessScanUnavailable path, which alerts after 12 consecutive ticks. The ValueError escaped to main(), which reported BrainBar activation monitor refused: {exc} immediately.
  • The scan auto-clear matched only the [Errno (one-shot legacy), >=12 ticks and Command '['/bin/ps', prefixes, so this reason latched until a human cleared it.

Fix (src/brainlayer/brainbar_auto_activation.py)

  • Tolerant parse: a 3-field row (empty args) is still checked by comm. A row without a decimal pid and a comm is skipped. Brew detection and the argv scrub are unchanged.
  • Zero parseable rows still raises ValueError from brew_processes, so the automatic activate child (brainbar_activation._activate -> brew_active(), which catches ValueError and boots out what it attempted) behaves as before. _defer_for_brew turns that ValueError into ProcessScanUnavailable("unreadable process table"), which is counted and never refuses immediately.
  • A successful scan now also clears BrainBar activation monitor refused: unreadable process table, so today's latched alert clears on the first good tick after install.

Test plan

  • Tests first: 5 new or parametrized cases failed on origin/main before the fix (5 failed, 42 passed).
  • pytest tests/test_brainbar_auto_activation.py tests/test_brainbar_activation.py: 76 passed.
  • ruff check + ruff format --check on both changed files are clean.
  • Pre-push BRAINLAYER_PREPUSH_SCOPE=changed-only: BrainLayer test gate passed.
  • Not run: the full suite (host load ~16), and a live tick against the installed keg. This needs the deploy hop and a real tick to confirm the latched alert clears.

Loop-record: implementer=claude/claude-opus-5-5/medium reviewer=none rounds=0/0 defects=0 role=worker
Review: pending Codex inner-loop review (routed by the BrainLayer P0 lead). No bot reviews requested.

— brainlayerClaude (worker) · claude-code/claude-opus-5-5

🤖 Generated with Claude Code


Note

Medium Risk
Changes BrainBar auto-activation gating and alert lifecycle during process scans; behavior is narrower (more tolerant) but still affects when activation runs and when critical alerts fire.

Overview
Fixes a latched brainbar-activation critical alert (BrainBar activation monitor refused: unreadable process table) caused when a single torn /bin/ps row during host stress made the whole table "unreadable."

brew_processes() now skips malformed rows (missing pid/comm, non-decimal pid) instead of failing the scan; three-field rows with empty args still participate in Homebrew detection via comm. ValueError is raised only when no parseable rows remain.

_defer_for_brew() maps that ValueError to ProcessScanUnavailable("unreadable process table"), so failures accumulate over ticks and alert only after 12 consecutive failures—matching other scan outages—rather than escaping to main() as an immediate monitor refusal. A successful scan also clears the latched "monitor refused: unreadable process table" alert.

Tests cover mixed valid/torn rows, wholly unusable tables (counted, not refused until tick 12), and auto-clear of the new refusal reason.

Reviewed by Cursor Bugbot for commit cac3054. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Fix brew_processes to tolerate torn rows in BrainBar activation scan

  • brew_processes now skips rows with fewer than three fields or an invalid PID, and treats a missing args field as empty. It raises ValueError only when no rows are readable.
  • AutoActivator._defer_for_brew converts that ValueError into a ProcessScanUnavailable deferral instead of an immediate refusal, and adds the unreadable-table message to the alert-recovery prefixes.
  • Behavioral Change: an unreadable process table now accumulates scan failures (alert after 11 ticks in tests) instead of refusing the scan; a later successful scan can clear the alert. Previously a single malformed row could invalidate the whole scan.

Macroscope summarized cac3054.

One malformed row from /bin/ps (a process exiting or exec-ing mid-scan)
raised ValueError("unreadable process table"), which escaped
_defer_for_brew and made main() latch an immediate
"BrainBar activation monitor refused" alert that nothing auto-cleared.

- brew_processes skips rows without a decimal pid and comm, and still
  checks 3-field (empty args) rows by comm.
- A table with zero parseable rows stays a ValueError for the automatic
  activate child, and _defer_for_brew counts it as ProcessScanUnavailable
  (alerted only after 12 ticks).
- A successful scan auto-clears the latched
  "monitor refused: unreadable process table" alert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EtanHey EtanHey added the size:S Tight-loop PR size: 51-150 hand-written lines added label Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 51 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 99acdc0f-de06-48f9-9935-fec3a2038b37
Commits

Reviewing files that changed from the base of the PR and between 1253f60 and cac3054.

Files selected for processing: 2
  • src/brainlayer/brainbar_auto_activation.py
  • tests/test_brainbar_auto_activation.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: b43593dc-08ac-4a00-82c0-daaea5dc8889)

@github-actions

Copy link
Copy Markdown

BrainLayer ratchet

Every Value below was measured by this run. A row this machine cannot measure says n/a — <reason> instead of a number; baselines in Notes name their own machine, method and date and were not measured here.

Hosted success gives no verdict on mapped bytes, search p50/p95, idle CPU, or fallback replay debt.

Row Status Value (measured by this run) Method Notes
commit provenance 🟢 GREEN measured cac30545201c == PR head · checkout cac30545201c commit graph + live PR head · in-process · runner Which commit this whole table is about. On a pull_request event the checkout is GitHub's synthetic merge ref, whose sha is not on the PR — #759's table printed 13fa724278bf while that PR's head was 4632f979 — so this row names the PR-head parent instead, the sha a reviewer can actually see. The comparison sha is read live from repos/{owner}/{repo}/pulls/{n} when the table is collected, not taken from the event payload, because the payload cannot know the run has been overtaken. Residual window, stated rather than papered over: a push landing between that read and the comment being posted is not caught here — the run for that push refreshes the table.
baseline attestation 🟢 GREEN baseline f421d1a7c5e6 matches the main attestation (run 40663077231 · main 1253f606dc59 · 2026-10-28T09:17:53Z) main attestation artifact via Actions API · in-process · runner What every comparison is measured AGAINST, and who says so. The baseline fields of tests/fixtures/sprint_gate/corpus.json (queries, latency_baseline_ms, thresholds) are compared to the ratchet-attestation artifact of the latest successful push or (no-input) workflow_dispatch run of ratchet-attest.yml on main, fetched through the Actions API — a PR run cannot write to another run's artifacts. A field that differs is RED unless that main run measured the new value. The calibrated socket collector can license p50/p95; every absent measured path stays locked, so missing collection never passes as permission for a hand edit. Boundary: the comparator is this PR's checkout of ci_ratchet_table.py, diff-reviewable, not tamper-proof.
provenance 🟢 GREEN stamped cac30545201c == HEAD, tree clean wheel stamp · in-process · runner Sha half of #749 keg-mode provenance: a keg built from this wheel can answer __build_sha__. The helper-age and served-process predicates need a running BrainBar and are measured only by scripts/sprint_gate.py on an installed Mac. The sha here is the checkout's — the explicit PR head in ratchet.yml — because that is what publish.yml stamps at release time; the PR-head sha this table describes is the one in commit provenance above.
fallback replay debt ⚪ n/a n/a — no fallback queue on this machine: the pending memories live in ~/Gits/*/docs.local/decisions, and docs.local/ is gitignored, so a runner checkout has no copy of them to count docs.local walk · machine with the fallback queue local-only; not enforced by hosted CI. intended_brain_store: true with no chunk_id means a memory reached disk and never reached the DB, so it answers no brain_search. Budget: 0. Any pending or unparseable file is a finding, never a band -- 122 of these sat from 2026-06-28 to 2026-09-05 because nothing counted them where a reader would look. Measured by walking the tree, so it is only ever measured on a machine that HAS the tree.
pre-push: delete-only push runs no suite 🟢 GREEN 0 suite starts · exit 0 real checkout hook · delete-only stdin · sentinel suite runner Bug SHA: 44ecbf5ba6f1ada3b965797559eee90c94b45595 (RED replay, 2026-10-06). Fix: #1084; measured at cac30545201cb29c1b7cf98bf8842f875435fd6c. Ceiling: 0 suite starts; hook must exit 0 and announce the delete-only skip.
mapped bytes ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would socket · installed Mac local-only; not enforced by hosted CI. Baseline 26.2 GB — installed Mac, socket, 2026-09-03, after R2 drained 15,070 → 0. Up from 16.8 GB because the drain left more vectors mapped under the same cap: the change is the drain, not a leak. Not measured by this run.
search p50/p95 ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would socket · installed Mac local-only; not enforced by hosted CI. Margin p50: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin p95: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Calibrated on MacBook-Pro.local at 2026-09-01T08:42:22Z under active_sprint_load (tests/fixtures/sprint_gate/corpus.json). Not measured by this run.
idle CPU ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would ps sampling · installed Mac local-only; not enforced by hosted CI. Ceiling: average CPU < 30% over a 60 s window (resource_budget in scripts/sprint_gate.py), ratified and kept as a hard budget. Margin daemon: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin helper: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin watcher: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Needs the BrainBar daemon, helper and watcher actually running. Not measured by this run.
signature_valid ⚪ n/a n/a — the macOS signature-parity job is trigger-gated and did not run on this PR: it touches no release or signing path (pyproject.toml, scripts/release-*, scripts/brainlayer-version-check.sh, publish.yml, ratchet.yml) and carries no ratchet:signatures label — a GitHub macOS runner bills at ~10× Linux minutes and rebuilds the keg venv from source codesign · installed keg scripts/release-verify-signatures.sh <keg> codesign-verifies every *.so/*.dylib under libexec/venv. The macOS parity job installs the published tap formula (etanhey/layers/brainlayer), so this row measures the release path — formula, published sdist and Homebrew's relocation — and not this PR's tree. Release-time baseline for the same keg on a different machine: 442 valid / 0 invalid — installed Mac (M4 Max), brew --prefix brainlayer 1.5.11, 2026-09-03.
BrainBar renders no enrichment status 🟢 GREEN 19 actual-view captures; active pending-store/replay retained macOS DEBUG binary · AppKit actual views · Vision OCR + icon RGBA comparison · private synthetic state Missing render capability/evidence is RED. Historical data is retained; this is source-build proof, not installed-app proof.
Watcher heartbeat freshness and recovery ⚪ n/a n/a — macOS native probe trigger-gated: no affected path or ratchet:heartbeat label private Python producer -> compiled Swift consumer Preserves genuine silence; independent agent stores cannot revive the watcher. Not a load or installed-UI measurement.
maintenance quiesce: services stay down under a reviving watchdog ⚪ n/a n/a — macOS replay trigger-gated: no affected path or ratchet:quiesce label real launchd · GitHub macOS runner · real maintenance + fleet-watchdog Fix: #1076 (bc6d690f4331387485678d0247e19e14e6ef8d97). Ceiling: 0 revivals; supervisor held, both resumed, no hold marker.
no cloud model call reachable anywhere 🟢 GREEN 508 source files; 444 installed imports; 3 native tests; wheel b7e95b3fff0b; 0 candidate attempts immutable installed wheel (default + dev) + AST source + actual native BrainBar Synthetic private fixtures; SDK absence gated on merged R10b. Exact-head candidate proof, not deployment.

🟢 GREEN measured, within budget · 🔴 RED measured, out of budget — a finding to clear before merge · ⚪ n/a not measurable on this machine, never guessed.

No RED rows.

Measured on Linux/x86_64 · measured cac30545201c · PR head cac30545201c · checkout cac30545201c · run · updated 2026-10-28 10:13:37 UTC

@EtanHey
EtanHey merged commit 1b00d06 into main Oct 28, 2026
26 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S Tight-loop PR size: 51-150 hand-written lines added

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant