From cbf40f73b8d5ba1878244f4a9e793400779d8e35 Mon Sep 17 00:00:00 2001 From: John Flavan Date: Tue, 27 Oct 2026 21:44:14 -0500 Subject: [PATCH 01/21] docs: design and plan for risky changes, retro, and acceptance tests Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-10-27-risk-retro-acceptance.md | 199 ++++++++++++++++++ ...2026-10-27-risk-retro-acceptance-design.md | 176 ++++++++++++++++ 2 files changed, 375 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-27-risk-retro-acceptance.md create mode 100644 docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md diff --git a/docs/superpowers/plans/2026-10-27-risk-retro-acceptance.md b/docs/superpowers/plans/2026-10-27-risk-retro-acceptance.md new file mode 100644 index 00000000..1429f39c --- /dev/null +++ b/docs/superpowers/plans/2026-10-27-risk-retro-acceptance.md @@ -0,0 +1,199 @@ +# Risky changes, retro, and acceptance tests: Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship three features: a merge gate that brings in a person for risky diffs, a `loopstra retro` report over the trace, and acceptance tests written from Done when before each build. + +**Architecture:** The three parts are independent and are built in parallel, each in its own worktree branched from `feat/risk-retro-acceptance`, then merged back. Part A (risk) changes the merge gate and the readers of a change's merge gate. Part B (retro) is a new read-only command over new Trace read methods. Part C (acceptance) adds a step to the build stage, a hook rule and a done-check variable. + +**Tech Stack:** Bun, TypeScript (strict), zod, bun:sqlite, bun:test, the fake `claude` and `gh` in `tests/`. + +**Spec:** `docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md`. Read it first; this plan does not repeat its wording rules or copy. + +## Global Constraints + +- Read `CLAUDE.md` and `docs/architecture.md` (layers, rules, TypeScript practices) before coding. Respect the layer order: a module imports only from its layer or below. +- `bun run check` (typecheck, lint, deadcode, tests) must pass before the part is done. During development run single test files: `bun test tests/unit/.test.ts`. +- Text a person reads is plain sentences: no stack traces, no raw commands. Owner notes never name branches except where the spec says. +- Every new JSON or frontmatter shape is a zod schema; types come from `z.infer`. +- Comments match the codebase: a short doc comment on each exported function saying what it is for, in plain words. +- No new dependencies. +- Each part adds its own README text and its own section to `docs/decisions.md` (append at the end, heading `## (2026-10-27)`), plus the template/skill text the spec lists for it. +- Commit in small steps on your part's branch with messages like `feat(risk): ...`, each ending with the line `Co-Authored-By: Claude Opus 5.5 `. +- Never run `npm publish` or touch release files; `package.json` stays `0.0.0-dev`. + +## Review Focus + +1. A risky change merged by the PR watcher before `merge_gate` is visible: the watcher reads the change's frontmatter fresh each look (ctx.intent), so the field must be written in the same record as the move to merge-review. Test: after passMergeGate on a risky change with `human: none` and a remote, one more merge step does not merge an unapproved PR. +2. Paths with spaces, non-ASCII names, renames (`old => new` in numstat) in the risk diff. Use `-z` output; test a renamed file into a risky path counts as risky. +3. `loopstra retro` on an empty or brand-new trace, and on a trace that has only `_chat` rows: prints the empty-window text, exits 0. +4. Acceptance step on a resumed build (status `building`, marker present): must not run twice. Test with the fake claude call log. +5. The hook on Windows paths (backslashes, case) for acceptance files: reuse the hook's existing `slashes`/`same` helpers; test a backslash path is refused. + +--- + +## Part A: Risky changes need a person (branch `feat/risk-gate`) + +### Task A1: Config and risk scoring + +**Files:** +- Modify: `src/config.ts` (gates.merge gains `risky`) +- Create: `src/risk.ts` (layer 2-3: imports `git.ts` and `config.ts` types only) +- Test: `tests/unit/risk.test.ts`, `tests/unit/config.test.ts` + +**Interfaces:** +- Produces: + - `gates.merge.risky: { paths: string[]; lines?: number; human: "status" | "pr" }` in `Config` (defaults `[]`, unset, `"pr"`). + - `export interface Risk { risky: boolean; files: string[]; lines: number; overLines: boolean }` + - `export function riskySet(cfg: Config): boolean` — true when `paths` is non-empty or `lines` is set. + - `export async function scoreRisk(wt: Git, main: string, cfg: Config): Promise` — numstat `-z` of `main...HEAD` in the worktree, files outside `intent/`, Bun.Glob match on each path (for renames, the new path), binary lines count 0. + - `export function riskEvidence(r: Risk): string` (gate row evidence: up to 10 files, then "and N more"; adds "N lines changed (over L)" when overLines). + - `export function riskNote(r: Risk, human: "status" | "pr"): string` (spec's note; at most 3 names then "and N more"). + +- [ ] Step 1: Config tests: `risky` defaults; `paths: [""]` → problem at `gates.merge.risky.paths.0`; `lines: 0` → problem; `human: none` → problem; unknown key under risky → unknown-key problem. +- [ ] Step 2: Risk tests in a throwaway repo (see `tests/helpers.ts` for repo helpers): no settings → not risky; `.github/**` matches `.github/workflows/ci.yml`; `package.json` matches only at top; `**/migrations/**` matches `db/migrations/1.sql`; `intent/x/intent.md` never counts; `lines: 5` with 6 changed lines → overLines; a binary file adds 0; a rename into `.github/` counts; a path with a space and one with `é` match. +- [ ] Step 3: Implement; run `bun test tests/unit/risk.test.ts tests/unit/config.test.ts` → pass. Commit. + +### Task A2: The effective merge gate everywhere + +**Files:** +- Modify: `src/intents.ts` (Frontmatter gains `merge_gate`, BLANK_LEFT_OUT gains it; helper), `src/actions.ts`, `src/attention.ts`, `src/stages/plan.ts`, any console reader that takes `human.merge` for one change (`src/console/model.ts`, `src/console/detail.ts` via `changeSummary`), `src/chat/orchestrator.ts` and `src/console/writes.ts` (they call `offers`) +- Test: `tests/unit/intents-frontmatter.test.ts`, `tests/unit/actions.test.ts`, `tests/unit/intents-queue.test.ts`, `tests/unit/status.test.ts` as fits + +**Interfaces:** +- Produces: + - Frontmatter `merge_gate: "status" | "pr" | undefined` (blank or absent reads as undefined; anything else is a frontmatter problem). + - `export function changeGates(human: HumanGates, fm: Frontmatter): HumanGates` in `src/intents.ts` — `human` with `merge` replaced by `fm.merge_gate` when `human.merge === "none"` and the field is set. + - `waitsForPerson`, `isRunnable`, `offers`, `personReviews` apply `changeGates` themselves (their signatures do not change), so every caller gets the per-change gate. + - `mayMergeFirst(cfg, fm)` in plan.ts takes the frontmatter. + +- [ ] Step 1: Tests: a `merge-review` change with `merge_gate: pr` and config `none` waits for a person, is not runnable without a remote, is offered Approve; without the field it is not waiting; config `status` with `merge_gate: pr` keeps `status`. +- [ ] Step 2: Implement; run the touched test files → pass. Commit. + +### Task A3: The merge stage decides and records + +**Files:** +- Modify: `src/stages/merge.ts`, `src/context.ts` only if `setStatus` needs a way to patch `merge_gate` in the same record (add an optional `fields` patch to `statusPatch`/`setStatus` rather than a second record) +- Test: `tests/unit/stages-review-merge.test.ts` (or a new `tests/integration/risk-gate.test.ts` following the integration tests' style) + +**Interfaces:** +- Consumes: `scoreRisk`, `riskySet`, `riskEvidence`, `riskNote` (A1); `changeGates` (A2). +- Produces: `function mergeHuman(ctx: StepContext, hasRemote: boolean): "none" | "status" | "pr"` (private): `changeGates(humanGates(ctx.cfg), ctx.intent.file.frontmatter).merge`, with `pr` read as `status` when there is no remote and the gate came from `merge_gate`. + +- [ ] Step 1: Tests: (a) remote, config none, risky change → status merge-review, `merge_gate: pr`, note per spec, gate row `merge/risk waiting`; a following merge step with checks passing and PR not approved waits; after approval it merges. (b) Same, not risky → no field, gate row `pass "nothing risky"`, merges on checks. (c) No remote, risky, `human: pr` → merge-review, `merge_gate: pr` acting as status: waits until merge-approved, then merges. (d) Config `status` + risky → gate row only, no field. (e) A change sent back and rebuilt without risky files → the field is removed at its next pass. +- [ ] Step 2: Implement in `passMergeGate` (score in the worktree after the checks pass; write the field and note in the same record as the status), and replace each `ctx.cfg.gates.merge.human` read in merge.ts with `mergeHuman`. +- [ ] Step 3: Run the stage tests and `tests/integration` files that touch merging → pass. Commit. + +### Task A4: Setup and docs + +**Files:** +- Modify: `src/setup/sections/github.ts` (ask for risky paths when `human` ends up `none`), `src/setup/defaults.ts` if it holds the defaults, `templates/config.yaml` (a commented `risky:` example under gates), README (GitHub, Configuration, When something needs a person), `templates/intent-README.md` (what `merge_gate` means), `docs/decisions.md` +- Test: `tests/unit/setup-github.test.ts` + +- [ ] Step 1: Tests: with `human: none`, the question is asked with the suggestion built from the files present (a repo with `package.json` and `go.mod` → `.github/**, **/migrations/**, package.json, go.mod`); Enter writes the list; `-` writes nothing; with `human: status` it is not asked. +- [ ] Step 2: Implement; run → pass. Docs. Commit. + +--- + +## Part B: `loopstra retro` (branch `feat/retro`) + +### Task B1: Trace reads + +**Files:** +- Modify: `src/trace.ts` +- Test: `tests/unit/trace.test.ts` + +**Interfaces:** +- Produces: + - `statusChangesSince(since: string): Array<{ slug: string; ts: string; from: string; to: string; note: string }>` (all slugs except `_chat`, oldest first; payload parsed through a zod schema) + - `eventsOfTypeSince(type: EventType, since: string): EventRow[]` + - `gatesSince(since: string): Array<{ slug: string; gate: string; check: string; result: string; ts: string }>` + - `phasesSince(since: string): PhaseRow[]` (started at or after `since`) + +- [ ] Step 1: Tests seed a Trace in a temp dir and check each method's filtering and order. Step 2: implement. Step 3: run → pass. Commit. + +### Task B2: The report + +**Files:** +- Create: `src/retro.ts` (pure: builds a `Retro` from rows, config and judge report; no I/O), `src/commands/retro.ts` (opens trace and config, renders text) +- Modify: `src/cli.ts` (command `retro`, `--days N`, usage line, help line "retro how the loop did lately, and what to change"), `src/context.ts`/stage files only to export the note constants retro classifies by +- Test: `tests/unit/retro.test.ts`, `tests/unit/cli.test.ts` + +**Interfaces:** +- Produces: + - `export type BlockKind = "timeout" | "crash" | "tests" | "review" | "verify" | "budget" | "conflicts" | "pr-checks" | "pr-closed" | "unmet" | "questions" | "other"` + - `export function blockKind(note: string): BlockKind` — by `includes` of exported note constants. + - `export function buildRetro(input: { since: string; now: Date; cfg: Config; changes: ...; blocks: ...; gates: ...; phases: PhaseRow[]; judges: JudgeReport; questions: EventRow[] }): Retro` + - `export function retroText(r: Retro): string` + - `export function suggestions(r: RetroFacts, cfg: Config): string[]` with the spec's rules and named threshold constants. + - `export async function renderRetro(root: string, days: number): Promise` in `src/commands/retro.ts`. + +- [ ] Step 1: Tests for `blockKind` on each real note constant and an unknown note. +- [ ] Step 2: Tests for `buildRetro` on seeded rows: counts, block grouping order, first-try pass rate per gate check, median accepted→merged, phase name folding (`fix-2`→fix, `early-review`→review, `spec-check-cheap`→spec-check), chat cost on its own line, person waits median. +- [ ] Step 3: Tests for each suggestion rule firing and not firing at its threshold; "No suggestions" text; empty window text. +- [ ] Step 4: CLI test: `retro --days 0` is a usage error; `retro --days 3` in a set-up repo prints the header "The last 3 days"; outside a set-up repo says NOT_SET_UP. +- [ ] Step 5: Implement; run → pass. Commit. + +### Task B3: Docs + +- [ ] README Watch (`loopstra retro`), `templates/skill/SKILL.md` (read how the loop is doing with `loopstra retro`), `docs/decisions.md`. Commit. + +--- + +## Part C: Acceptance tests (branch `feat/acceptance-tests`) + +### Task C1: Config, step, envelope, prompt + +**Files:** +- Modify: `src/config.ts` (`stages.build.acceptance_tests: boolean`, default false), `src/models.ts` (STEP_KEYS + STEPS entry `acceptance`, group build, "writes tests from Done when before the build", follows build; place after `plan-challenge`), `src/envelopes.ts` (`acceptance` envelope), `src/console/steerable.ts` (steerable), `templates/prompts/acceptance.md` (new), `templates/config.yaml` (`acceptance_tests: true` on the build line), `src/setup/sections/gates.ts` (the yes/no question, suggestion yes), anything that enumerates prompts or steps (`src/prompts.ts`, `src/shipped.ts`, `src/upgrade/*`, console settings page lists) — find them with `grep -rn '"reconcile"' src` +- Test: `tests/unit/config.test.ts`, `tests/unit/models.test.ts`, `tests/unit/envelopes.test.ts`, `tests/unit/setup-gates.test.ts`, `tests/unit/templates.test.ts` + +**Interfaces:** +- Produces: `Envelopes.acceptance = base.extend({ commit_message: z.string(), tests: z.array(z.object({ criterion: z.string(), file: z.string(), test: z.string() })), not_testable: z.array(z.object({ criterion: z.string(), why: z.string() })) })`. Prompt variables: `done_when`, `spec`, `plan`, `test_command`, `conversation` (as work sessions get). + +- [ ] Step 1: Tests: config default false and type error text; `stepModel(cfg, "acceptance")` follows build; envelope parses the spec's example; setup gates asks and writes `stages.build.acceptance_tests`; the shipped prompt renders with all its variables. +- [ ] Step 2: Implement; run → pass. Commit. + +### Task C2: The hook + +**Files:** +- Modify: `templates/hooks/loopstra-protect-tests.ts` +- Test: `tests/unit/hook.test.ts` + +**Interfaces:** +- Consumes: env `LOOPSTRA_PHASE`, `LOOPSTRA_ACCEPTANCE` (JSON array of repo-relative paths; unreadable JSON reads as none). +- Behaviour: phase `acceptance` → refuse any path that is not a test path (message: "Loopstra: the acceptance step writes only tests and test fixtures ()."). Phases `build`, `fix`, `reconcile`, `revise` → refuse a path in `LOOPSTRA_ACCEPTANCE` (logged to `LOOPSTRA_PROTECTED_LOG`; message per spec). The existing fix-phase rule is unchanged. + +- [ ] Step 1: Tests for each rule, plus a backslash/case variant and malformed JSON. Step 2: implement. Step 3: run → pass. Commit. + +### Task C3: The build runs the step and protects the files + +**Files:** +- Modify: `src/stages/build.ts`, `src/stages/shared.ts` (marker name `ACCEPTANCE = "acceptance"`; `acceptanceFiles(ctx): string[]` reading it), `src/phases.ts` (env `LOOPSTRA_ACCEPTANCE` for every session from the marker; `protectedNote` words acceptance files per spec), `tests/fake-claude/fixtures/acceptance.jsonl` (writes `tests/accept.test.ts`, answers one test and one not-testable item) +- Test: `tests/unit/stages-build.test.ts`, `tests/unit/phases.test.ts` + +**Interfaces:** +- Produces: `async function writeAcceptance(ctx, wt): Promise` (private to build.ts): runs the phase, commits, writes the marker (JSON array of files), records `acceptance.md` via `writeArtifact`. `acceptance.md` is added to the artifacts the console lists (`ARTIFACTS` in `src/intents.ts`). + +- [ ] Step 1: Tests: setting off → no acceptance call; on → the call happens before build (fake call log order), the commit `loopstra(): acceptance tests` is on the branch, `acceptance.md` on main matches the spec's layout, the build session's env has the file; a resumed build with the marker makes no second call; a new build from plan-approved clears and reruns; a failing acceptance session → trace error, acceptance.md says no tests were written, build goes on; drift leaves the acceptance file out (no reconcile call); a fix refused on the acceptance file blocks with the spec's note naming it. +- [ ] Step 2: Implement; run → pass. Commit. + +### Task C4: Done-check reads acceptance.md + +**Files:** +- Modify: `src/stages/verify.ts` (var `acceptance`; appended through `phasePrompt`'s missing-variable note when a customized prompt lacks it), `templates/prompts/done-check.md` +- Test: `tests/unit/stages-verify.test.ts`, `tests/unit/prompts.test.ts` + +- [ ] Step 1: Tests: the done-check prompt contains the Covered list when acceptance.md exists, and "No acceptance tests were written for this change." when not; a customized done-check.md without the variable still gets the text. Step 2: implement. Step 3: run → pass. Commit. + +### Task C5: Docs + +- [ ] README (How it works, Configuration, the step table row `acceptance | build | the build stage's model`, Steering's list of work sessions), `templates/intent-README.md` (acceptance.md), `docs/decisions.md`. Commit. + +--- + +## Integration (after the three parts) + +- [ ] Merge `feat/risk-gate`, `feat/retro`, `feat/acceptance-tests` into `feat/risk-retro-acceptance`; resolve `docs/decisions.md`, README and `src/config.ts` conflicts. +- [ ] Retro's suggestions that name `gates.merge.risky` and `stages.build.acceptance_tests` work against the merged config (Part B writes them against those exact key names). +- [ ] `bun run check` → pass. `/code-review high`, fix, PR, CI green. diff --git a/docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md b/docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md new file mode 100644 index 00000000..f51a9a09 --- /dev/null +++ b/docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md @@ -0,0 +1,176 @@ +# Risky changes, `loopstra retro`, and acceptance tests + +2026-10-27. Three features that make the loop safer to leave alone and easier to tune. The user chose them from a list of ideas and asked Claude to make every decision; each decision below is Claude's call, recorded in `docs/decisions.md` when it is built. + +## Why + +Loopstra's goal is a loop that runs unattended, stays simple, and keeps human-readable records. Today a person picks between two blunt settings: a person on every merge (slow), or nobody on any merge (fast, but a change to CI, dependencies or migrations goes in unseen). The done-check that decides whether a change did what the owner asked is an agent's judgment, made after the merge. And the trace records everything, but nothing reads it back to say how the loop is doing or what to change. + +1. **Risky changes need a person.** Merges stay unattended, except for changes that touch files an engineer named as risky. Those wait for a person, whatever `gates.merge.human` says. +2. **`loopstra retro`.** A plain report over the trace for the last days: what shipped, where changes blocked, how the gates did, what it cost, how long people kept changes waiting, and concrete config suggestions with their evidence. +3. **Acceptance tests from Done when.** Before the build, a separate session writes tests from the owner's Done when list. The build may not change them, so they hold it to what was asked, and the done-check reads them as evidence. + +## Goals + +- Each feature is off unless configured, or (acceptance tests) on only for repos set up after it ships, so an upgrade changes no running loop's behaviour. +- Every decision is in a file a person can read: the intent's frontmatter and note, a gate row in the trace, `acceptance.md`. +- No new long-running process, no new storage. Retro reads the trace that exists. + +## Non-goals + +- No agent judging risk. Risk is facts about the diff (paths, size), so it is predictable and testable. +- Retro does not change the config. It suggests; a person edits. +- Retro is not posted to chat or shown in the console in this change (a follow-up). +- Acceptance tests do not replace the project's own test suite or the verifier. + +## Anti-goals + +- A risky change must never merge with nobody looking because of a stale or missing record. When in doubt, a person decides. +- Acceptance tests must never leave a change stuck with no way out: a wrong acceptance test has to be named in the block note with what to do. + +--- + +## 1. Risky changes need a person + +### Config + +```yaml +gates: + merge: + human: none + method: squash + risky: + paths: [".github/**", "package.json", "bun.lock", "**/migrations/**"] + lines: 1500 + human: pr +``` + +- `risky.paths`: glob patterns (Bun.Glob, matched against repo-relative paths with `/`). Default `[]`. A bare name like `package.json` matches only at the top; `**/package.json` matches anywhere. Each entry must be a non-empty string. +- `risky.lines`: optional positive whole number. A change whose added plus deleted lines (outside `intent/`, binary files not counted) exceed it is risky. Unset: size never makes a change risky. +- `risky.human`: `pr` (default) or `status`: who decides a risky change. Without a GitHub remote, `pr` acts as `status`: there is no pull request, so a person sets `merge-approved`. (Claude's call: a risky change in a repo without a remote should wait, not block.) + +Nothing is risky when `paths` is empty and `lines` is unset, which is the default, so existing configs behave as before. + +### When it is decided + +At the one place the merge gate passes its automated checks, `passMergeGate` in `src/stages/merge.ts`. Every change Loopstra makes to a branch (build, fix, revise, a fix during the merge checks) goes through another review and then through this point, so the decision is made again on the code that will merge. Commits a person pushes to the pull request on GitHub are a person's work and are not scored again (Claude's call: simpler, and a person is already involved). + +The diff is the branch against main (merge base), outside `intent/`: `git diff --numstat -z
...` in the worktree. + +### What it records + +- A gate row: gate `merge`, check `risk`, result `pass` with evidence "nothing risky" (only when risky settings exist), or `waiting` with evidence naming the matched files (up to 10) and the line count when that tripped it. +- Frontmatter field `merge_gate: pr | status` on the change, written by the runtime in the same record as the status move to `merge-review`, and removed by that record when the change is not risky. Runtime-managed, left out when blank (like `hold`). It is only written when it changes who decides, that is when `gates.merge.human` is `none`. When the config already has a person on the merge gate, the gate row is still written and no field is. +- The note says why, in plain words: "This change touches files set to need a person before merging: .github/workflows/ci.yml and package.json. Approve the pull request on GitHub, or set status to merge-approved." (With `status`: "Read review.md. To let this change in, change the status line to merge-approved." with the same first sentence.) File names are allowed in this note (Claude's call): whoever approves a merge needs to know what to look at. At most 3 names, then "and N more". + +### Who reads it + +One helper decides the effective merge gate of a change: `gates.merge.human` when it is not `none`, else the change's `merge_gate`, else `none`. The readers that today use `human.merge` or `cfg.gates.merge.human` for a specific change use it instead: + +- `waitsForPerson`, `isRunnable` (`src/intents.ts`), so the scheduler does not run a waiting risky change and `status`/`queue.md`/attention list it as waiting for a person. +- `offers`/`personReviews` (`src/actions.ts`), so the console and chat offer Approve and Send back. +- `attention.ts`, for its note. +- `src/stages/merge.ts`: `passMergeGate`, `watchPullRequest` (wait for an approval on GitHub, or for `merge-approved`), `runMergeStep` (no remote: merge-review waits), `mergeRetry`. +- `mayMergeFirst` in `src/stages/plan.ts` (a risky change in merge-review waits for a person, so it does not merge first). + +A person who deletes the `merge_gate` line takes the change back to the config's gate; a person's edits win, as everywhere. + +### Setup + +`loopstra setup gates` asks, when the merge gate has nobody on it: "Files whose change needs a person before merging (comma-separated globs)". The suggestion is `.github/**` and `**/migrations/**`, plus each of these that exists at the top of the repo: `package.json`, `bun.lock`, `package-lock.json`, `pnpm-lock.yaml`, `yarn.lock`, `go.mod`, `go.sum`, `Cargo.toml`, `Cargo.lock`, `pyproject.toml`, `requirements.txt`, `Gemfile`, `Gemfile.lock`, `Dockerfile`. `-` leaves the list empty. `lines` and `human` are left to the file (and documented). + +--- + +## 2. `loopstra retro` + +``` +loopstra retro # the last 7 days +loopstra retro --days 30 +``` + +Read-only. Plain text, like `loopstra judges`. Reads `.loopstra/trace.db` and the config. Leaves out `_chat` except in the spending line, where chat's cost is shown on its own. + +### Sections + +1. **Header.** "The last 7 days (since Oct 20)." +2. **What happened.** Changes that reached main (status moved to `merged`), finished (`done`), blocked (moved to `blocked`, counted once per change), closed. The median time from `accepted` (or the first status after it) to `merged`, for changes that got there in the window. +3. **Where changes blocked.** Each block in the window, grouped by kind, most common first, with the changes named: timed out, the assistant crashed, tests kept failing, the reviewer kept finding problems, the verifier found it did not work, the spending limit, merge conflicts, checks on GitHub failed, the pull request was closed, a Done when criterion not met, a question limit, and other (with its note). The kind comes from the block note: each recognized note is matched by the constant the runtime writes it from, exported where it is local today, never by a fresh regex on wording. +4. **Gates.** One line per gate check that ran in the window (spec check, plan challenge, review findings, merge risk, pr-checks, done-check): how many times it ran, how many changes passed it the first time. From the `gates` table. +5. **Spending.** Total for the loop, then by kind of step (phase names folded: `fix-2`, `test-3` → `fix`, `test`; `early-review` counts as review; `-cheap` as its judge), the five biggest, with the share of the total and the average per change. Chat on its own line. +6. **Waiting for people.** For each review status a person decides (spec-review, plan-review, merge-review, verifying) and for open questions: how many times a change waited there and the median wait. From `status_change` and `question` events. +7. **Suggestions.** Zero or more, each one sentence with its evidence and the setting to change. Rules (thresholds are named constants): + - Cheaper judges: the setting is `shadow` and a judge's record has earned going first (`judgeReport`'s `goesFirst`): suggest `speed.cheap_judges: first`. The setting is `off` and judges cost at least 30% of the window's spending over at least 5 changes: suggest `shadow`. + - Fix loops: at least 2 blocks of "tests kept failing": suggest `stages.build.max_fix_loops` one higher, or smaller changes. + - Timeouts: at least 2 timeouts: suggest `claude.timeout_minutes` 50% higher (rounded to 5), or smaller changes. + - Review rounds: at least 2 blocks of "reviewer still found": suggest `stages.review.max_rounds` one higher. + - Spending limit: at least 1 block on the session's limit: suggest raising `claude.max_budget_usd` (with `loopstra setup budgets`). + - Waits: a person-decided gate whose median wait is over 24 hours, where the automatic check of that gate passed every time in the window (at least 5): suggest the person could come off that gate. + - Risky changes: `gates.merge.human: none` and nothing set under `gates.merge.risky`, with at least 1 change merged: suggest setting `risky.paths`. + - Acceptance tests: off, and at least 1 change blocked on a Done when criterion not met: suggest `stages.build.acceptance_tests: true`. + - None apply: "No suggestions: nothing in the trace points at a setting to change." + +An empty window: "Nothing happened in the last 7 days." and the suggestions that apply anyway (none of the rules above fire without data). + +### Trace queries + +New read methods on `Trace` (no schema change): status changes since a time (across slugs), gate rows since a time, phases started since a time. Indexes exist for phases by start; events are filtered by type and time. The trace is small enough that a scan of events of one type is fine. + +--- + +## 3. Acceptance tests from Done when + +### Config + +`stages.build.acceptance_tests: true | false`. The schema default is `false`, so an upgraded repo is unchanged. `templates/config.yaml` (what `loopstra init` writes) sets it to `true`, and `loopstra setup gates` asks "Write tests from Done when before each build? (Y/n)". (Claude's call: new repos get it, old repos opt in; the step costs one more session per change.) + +A new step key `acceptance` in `STEPS` (group `build`, "writes tests from Done when before the build", model follows the build stage), so it has a model setting and an `agents.steps.acceptance` setting like the others. + +### The step + +At the start of `runBuildStep`, after the worktree is ready and the build's `before` commands ran, and before the build session, when `acceptance_tests` is on and this build has not written its acceptance tests yet (run-folder marker `acceptance`): + +1. A fresh agent session `acceptance` in the change's worktree, `acceptEdits`, the build's tools, with the Done when list, the spec, the plan and the test command. It writes tests in the project's own style and test folders, one or more per Done when item it can test, runs them, and expects them to fail before the build. It must not change anything but tests and test fixtures. The test-protection hook enforces that in this phase: only paths it considers tests may be edited. +2. Its answer: `tests: [{ criterion, file, test }]` (the criterion in the owner's words, the file, the test's name) and `not_testable: [{ criterion, why }]`, plus `summary` and `commit_message`. +3. The runtime commits what it wrote on the branch (`loopstra(): acceptance tests`), writes the marker (the list of test files), and records `intent//acceptance.md` on main: + + ```markdown + # Acceptance tests + + Written from Done when before the build. The build may not change them. + + ## Covered by tests + - Counting the words in "hello world" gives 2: `tests/count.test.ts`, "counts two words" + + ## Not covered by a test + - People find the new page easy to read: needs a person. + ``` + +4. A session that fails, or writes no test at all, does not block the change (Claude's call): the trace says why, acceptance.md says no tests were written, and the build goes on as without the setting. Acceptance tests are a help, not a gate. + +A new build (status was not `building`: from `plan-approved`, a send-back) clears the marker, so its acceptance step runs again on the branch and may update what an earlier build's step wrote. A build resumed after a stop keeps the marker and does not run the step twice. + +### Protection + +Every session of the build that may edit (build, fix, reconcile, revise, also the fixes during the merge checks) gets the change's acceptance test files in `LOOPSTRA_ACCEPTANCE` (a JSON list, from the marker). The hook refuses edits to those files in those phases, logs them like today's refused tests, and says "this is an acceptance test written from Done when; fix the code, not the test". Today's rule (the fix phase may not edit tests the branch did not add) stays as it is. + +When a session fails after the hook refused an acceptance test, the block note names it: "It was not allowed to edit the acceptance test `tests/count.test.ts`, written from Done when before the build. If the test is wrong, an engineer can fix it on the branch intent/count-words." The usual retry advice follows. + +The drift check (files changed that the plan did not list) leaves the acceptance test files out, so they do not start a reconcile session. + +### The done-check + +The done-check's prompt gets `{{acceptance}}`: acceptance.md's "Covered by tests" list, or "No acceptance tests were written for this change." It is told: a criterion covered by an acceptance test that passes in this copy of main is met, with the test as its evidence. A customized `done-check.md` without the variable still gets the text appended, the way `{{feedback}}` reaches an older `build.md`. + +### Steering + +`acceptance` is a work session: it can be messaged, redirected and stopped like build. It does not ask questions (Claude's call: the Done when list is the owner's answer already). + +--- + +## Docs and tests + +- README: the merge gate's risky settings under GitHub and Configuration, `merge_gate` under "When something needs a person", `loopstra retro` under Watch, acceptance tests under How it works and Configuration, the new step in the models table. +- `templates/intent-README.md` (owners): what `merge_gate` and acceptance.md mean. +- `templates/skill/SKILL.md`: `loopstra retro` as a way to read how the loop is doing. +- `docs/decisions.md`: one section with each call above. +- Tests: config schema (each new key and its errors); risk scoring (globs, lines, binary files, intent/ left out); the effective gate in every reader; the merge step with and without a remote for a risky and a non-risky change; retro's grouping, medians and each suggestion rule, on a seeded trace; the hook's acceptance rule; the acceptance step (writes, commits, records, skips on resume, fails softly), protection and the block note; drift leaving acceptance files out; the done-check's variable; setup's new questions. From 3f859a3354b42f3ae216bb59b5aceeaaaf2faf3e Mon Sep 17 00:00:00 2001 From: John Flavan Date: Tue, 27 Oct 2026 21:48:06 -0500 Subject: [PATCH 02/21] feat(retro): trace reads since a time: status moves, events of a type, gate rows, phases Co-Authored-By: Claude Opus 5.5 --- src/trace.ts | 43 +++++++++++++++++++++++ tests/unit/trace.test.ts | 75 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 118 insertions(+) diff --git a/src/trace.ts b/src/trace.ts index 74ed1c54..ae01ed13 100644 --- a/src/trace.ts +++ b/src/trace.ts @@ -1,6 +1,8 @@ import { Database } from "bun:sqlite"; import { appendFileSync, existsSync, mkdirSync, renameSync } from "node:fs"; import { basename, join } from "node:path"; +import { z } from "zod"; +import { CHAT_SLUG } from "./budget"; import type { ModelRef } from "./models"; import { loopstraDir, runDir } from "./paths"; import { errorCode, errorText } from "./shell"; @@ -65,6 +67,15 @@ export interface ModelRun { ref: ModelRef; asked: string; ran: string | null; result: "ok" | "bad-model"; slug: string; seq: number; ts: string; } +/** A status_change event's payload, as statusChange writes it (other fields, like who moved it, are not read here). */ +const StatusPayload = z.object({ from: z.string(), to: z.string(), note: z.string().default("") }); + +/** One status move of a change: when, from what, to what, and the note it left. */ +export interface StatusMove { slug: string; ts: string; from: string; to: string; note: string } + +/** One gate check's result, without its evidence. */ +export interface GateRow { slug: string; gate: string; check: string; result: string; ts: string } + /** A judges row as written: the trace adds the id, the change and the time. */ export type NewJudgeRow = Omit; @@ -544,6 +555,38 @@ export class Trace { return this.db.query<{ c: number | null }, string[]>(`SELECT SUM(cost_usd) AS c FROM phases WHERE ${who} AND started >= ?${running}`).get(...args)?.c ?? 0; } + /** + * Every status move of every change (chat's left out) at or after `since` (an ISO time), oldest + * first. A row whose payload cannot be read is passed over. + */ + statusChangesSince(since: string): StatusMove[] { + const rows = this.eventsOfTypeSince("status_change", since); + const out: StatusMove[] = []; + for (const r of rows) { + if (r.slug === CHAT_SLUG) continue; + let raw: unknown; + try { raw = JSON.parse(r.payload); } catch { continue; } + const p = StatusPayload.safeParse(raw); + if (p.success) out.push({ slug: r.slug, ts: r.ts, ...p.data }); + } + return out; + } + + /** Every event of one type at or after `since` (an ISO time), of every slug, oldest first. */ + eventsOfTypeSince(type: EventType, since: string): EventRow[] { + return this.db.query("SELECT * FROM events WHERE type = ? AND ts >= ? ORDER BY id").all(type, since); + } + + /** Every gate check's result at or after `since` (an ISO time), of every change, oldest first. */ + gatesSince(since: string): GateRow[] { + return this.db.query('SELECT slug, gate, "check", result, ts FROM gates WHERE ts >= ? ORDER BY id').all(since); + } + + /** Every phase (chat's too) started at or after `since` (an ISO time), in the order they started. */ + phasesSince(since: string): PhaseRow[] { + return this.db.query("SELECT * FROM phases WHERE started >= ? ORDER BY started, rowid").all(since); + } + /** What a slug's ended phases cost, in all (a running phase's budget hold is not spending). */ endedCost(slug: string): number { return this.db.query<{ c: number | null }, [string]>("SELECT SUM(cost_usd) AS c FROM phases WHERE slug = ? AND status != 'running'").get(slug)?.c ?? 0; diff --git a/tests/unit/trace.test.ts b/tests/unit/trace.test.ts index b7d7cb2c..50b63fe6 100644 --- a/tests/unit/trace.test.ts +++ b/tests/unit/trace.test.ts @@ -382,3 +382,78 @@ describe("two connections at once", () => { } finally { t.cleanup(); } }); }); + +describe("reads since a time, for loopstra retro", () => { + /** Sets the time of every row of `table` by its id (rowid for phases), oldest given first. */ + function stampAll(root: string, table: "events" | "gates" | "phases", times: string[]): void { + const db = new Database(join(root, ".loopstra", "trace.db")); + try { + const key = table === "phases" ? "rowid" : "id"; + const column = table === "phases" ? "started" : "ts"; + times.forEach((ts, i) => db.run(`UPDATE ${table} SET ${column} = ? WHERE ${key} = ?`, [ts, i + 1])); + } finally { db.close(); } + } + + const DAY1 = "2026-10-01T10:00:00.000Z"; + const DAY2 = "2026-10-02T10:00:00.000Z"; + const DAY3 = "2026-10-03T10:00:00.000Z"; + + test("status changes since a time: every change but chat, oldest first, with the payload read", () => { + const t = tempDir(); + const trace = Trace.open(t.path); + try { + trace.statusChange("a-b", "accepted", "designing"); + trace.statusChange("c-d", "building", "blocked", "The tests kept failing.", { retry: "plan-approved" }); + trace.statusChange("_chat", "x", "y"); + trace.statusChange("a-b", "designing", "spec-review", "Read spec.md."); + trace.event("a-b", "status_change", "not an object"); + trace.event("a-b", "tick", {}); + stampAll(t.path, "events", [DAY1, DAY2, DAY2, DAY3, DAY3, DAY3]); + expect(trace.statusChangesSince(DAY2)).toEqual([ + { slug: "c-d", ts: DAY2, from: "building", to: "blocked", note: "The tests kept failing." }, + { slug: "a-b", ts: DAY3, from: "designing", to: "spec-review", note: "Read spec.md." }, + ]); + expect(trace.statusChangesSince(DAY1)).toHaveLength(3); + } finally { trace.close(); t.cleanup(); } + }); + + test("events of one type since a time, oldest first", () => { + const t = tempDir(); + const trace = Trace.open(t.path); + try { + trace.event("a-b", "question", { question: "old" }); + trace.event("a-b", "tick", {}); + trace.event("c-d", "question", { question: "new" }); + stampAll(t.path, "events", [DAY1, DAY2, DAY3]); + expect(trace.eventsOfTypeSince("question", DAY2).map((e) => [e.slug, e.payload])).toEqual([["c-d", '{"question":"new"}']]); + expect(trace.eventsOfTypeSince("question", DAY1)).toHaveLength(2); + } finally { trace.close(); t.cleanup(); } + }); + + test("gate rows since a time, oldest first", () => { + const t = tempDir(); + const trace = Trace.open(t.path); + try { + trace.gate("a-b", "spec", "spec-check", "fail", "two findings"); + trace.gate("a-b", "spec", "spec-check", "pass", "ok"); + trace.gate("c-d", "merge", "pr-checks", "waiting", "running"); + stampAll(t.path, "gates", [DAY1, DAY2, DAY3]); + expect(trace.gatesSince(DAY2)).toEqual([ + { slug: "a-b", gate: "spec", check: "spec-check", result: "pass", ts: DAY2 }, + { slug: "c-d", gate: "merge", check: "pr-checks", result: "waiting", ts: DAY3 }, + ]); + } finally { trace.close(); t.cleanup(); } + }); + + test("phases started at or after a time, in the order they started, chat's too", () => { + const t = tempDir(); + const trace = Trace.open(t.path); + try { + trace.phaseStart("a-b", "design", "agent"); + trace.phaseStart("_chat", "chat", "agent"); + trace.phaseStart("a-b", "spec-check", "agent"); + stampAll(t.path, "phases", [DAY1, DAY3, DAY2]); + expect(trace.phasesSince(DAY2).map((p) => [p.slug, p.name])).toEqual([["a-b", "spec-check"], ["_chat", "chat"]]); + } finally { trace.close(); t.cleanup(); } + }); +}); From 604aa7b273cd67febdee7959cbbe3711923bdbb7 Mon Sep 17 00:00:00 2001 From: John Flavan Date: Tue, 27 Oct 2026 21:49:44 -0500 Subject: [PATCH 03/21] feat(acceptance): config, step, envelope and prompt for acceptance tests stages.build.acceptance_tests (off by default; init's template turns it on), the acceptance step under build, its envelope and prompt, a steerable work session, and the yes/no question in setup gates. Co-Authored-By: Claude Opus 5.5 --- src/agents.ts | 2 +- src/config.ts | 7 +++++- src/console/steerable.ts | 2 +- src/envelopes.ts | 9 +++++++ src/models.ts | 3 ++- src/setup/sections/gates.ts | 7 +++++- templates/config.yaml | 4 +++- templates/prompts/acceptance.md | 29 +++++++++++++++++++++++ tests/unit/agents.test.ts | 3 ++- tests/unit/config.test.ts | 11 +++++++++ tests/unit/console-agent-settings.test.ts | 2 +- tests/unit/envelopes.test.ts | 14 ++++++++++- tests/unit/models.test.ts | 5 ++-- tests/unit/setup-gates.test.ts | 28 ++++++++++++++++++---- tests/unit/templates.test.ts | 12 ++++++++++ 15 files changed, 123 insertions(+), 15 deletions(-) create mode 100644 templates/prompts/acceptance.md diff --git a/src/agents.ts b/src/agents.ts index 5ff94ecd..95994ce7 100644 --- a/src/agents.ts +++ b/src/agents.ts @@ -12,7 +12,7 @@ export const TEAM_WAYS = ["parallel", "pick"] as const; export type TeamWay = (typeof TEAM_WAYS)[number]; /** Steps that change files: the rest are read-only. */ -const WRITING_STEPS: readonly StepKey[] = ["build", "reconcile", "fix", "revise"]; +const WRITING_STEPS: readonly StepKey[] = ["acceptance", "build", "reconcile", "fix", "revise"]; /** Every step except those that change files. */ export const READ_ONLY_STEPS: ReadonlySet = new Set(STEP_KEYS.filter((k) => !WRITING_STEPS.includes(k))); diff --git a/src/config.ts b/src/config.ts index 74cd7c7a..01115797 100644 --- a/src/config.ts +++ b/src/config.ts @@ -212,7 +212,12 @@ export const ConfigSchema = z.object({ stages: z.object({ design: stage("design").extend({ model: modelRef.default("strong") }).prefault({}), plan: stage("plan").extend({ model: modelRef.default("strong") }).prefault({}), - build: stage("build").extend({ model: modelRef.default("default"), max_fix_loops: z.number().int().min(1).default(3) }).prefault({}), + build: stage("build").extend({ + model: modelRef.default("default"), + max_fix_loops: z.number().int().min(1).default(3), + /** A session writes tests from the owner's Done when list before each build, and the build may not change them. Off unless set, so an upgraded repo is unchanged; `loopstra init` turns it on. */ + acceptance_tests: z.boolean({ error: "must be true or false" }).default(false), + }).prefault({}), review: stage("review").extend({ model: modelRef.default("strong"), max_rounds: z.number().int().min(1).default(2) }).prefault({}), verify: stage("verify").extend({ model: modelRef.default("default") }).prefault({}), }).strict().prefault({}), diff --git a/src/console/steerable.ts b/src/console/steerable.ts index 3f24f111..40ab2868 100644 --- a/src/console/steerable.ts +++ b/src/console/steerable.ts @@ -4,7 +4,7 @@ */ /** The work sessions a person may message or redirect (any session may be stopped). Shared by the loop and the page. */ -const STEERABLE_PHASES = ["intake", "design", "plan", "build", "fix", "reconcile", "revise", "lessons"] as const; +const STEERABLE_PHASES = ["intake", "design", "plan", "acceptance", "build", "fix", "reconcile", "revise", "lessons"] as const; /** * A phase row's name as the phase it runs: "fix-2" → fix, "early-lessons" → lessons, "design-retry" → design, diff --git a/src/envelopes.ts b/src/envelopes.ts index e719d519..8bc65956 100644 --- a/src/envelopes.ts +++ b/src/envelopes.ts @@ -53,6 +53,15 @@ export const Envelopes = { approved: z.boolean(), concerns: z.array(z.object({ concern: z.string(), blocking: z.boolean() })), }), + /** + * Tests written from the owner's Done when list before the build: each test by the criterion it covers + * (in the owner's words), its file and its name, and each criterion no test could cover, with why. + * It never asks: the Done when list is the owner's answer already. The files it protects come from git. + */ + acceptance: codeChange.extend({ + tests: z.array(z.object({ criterion: z.string(), file: z.string(), test: z.string() })), + not_testable: z.array(z.object({ criterion: z.string(), why: z.string() })), + }), build: askingCodeChange, fix: codeChange, reconcile: base.extend({ plan_markdown: z.string() }), diff --git a/src/models.ts b/src/models.ts index 264afd00..3417875d 100644 --- a/src/models.ts +++ b/src/models.ts @@ -11,7 +11,7 @@ export type StageName = (typeof STAGE_NAMES)[number]; export type StepGroup = StageName | "chat"; export const STEP_KEYS = [ - "intake", "design", "spec-check", "plan", "plan-challenge", "build", "reconcile", "fix", + "intake", "design", "spec-check", "plan", "plan-challenge", "acceptance", "build", "reconcile", "fix", "review", "revise", "verify", "done-check", "lessons", "orchestrator", "write-intent", ] as const; export type StepKey = (typeof STEP_KEYS)[number]; @@ -34,6 +34,7 @@ export const STEPS: readonly StepInfo[] = [ { key: "spec-check", group: "design", what: "judges the spec", byDefault: { ref: "strong" } }, { key: "plan", group: "plan", what: "writes the plan (also the plan written ahead)", byDefault: { follows: "plan" } }, { key: "plan-challenge", group: "plan", what: "judges the plan", byDefault: { ref: "strong" } }, + { key: "acceptance", group: "build", what: "writes tests from Done when before the build", byDefault: { follows: "build" } }, { key: "build", group: "build", what: "writes the code", byDefault: { follows: "build" } }, { key: "reconcile", group: "build", what: "reconciles after the base moved", byDefault: { follows: "build" } }, { key: "fix", group: "build", what: "fixes after failed checks, and after a failed verify", byDefault: { follows: "build" } }, diff --git a/src/setup/sections/gates.ts b/src/setup/sections/gates.ts index 79be35ac..cf64a830 100644 --- a/src/setup/sections/gates.ts +++ b/src/setup/sections/gates.ts @@ -12,7 +12,7 @@ const GATES = [ export const gates: Section = { name: "gates", title: "Gates", - covers: ["gates.spec", "gates.plan", "gates.done"], + covers: ["gates.spec", "gates.plan", "gates.done", "stages.build.acceptance_tests"], async ask(ctx) { ctx.ask.say("A person always accepts a change (draft to accepted). After that, each step can wait for a person or go on by itself. The merge gate is under github."); @@ -24,6 +24,11 @@ export const gates: Section = { const agent = await ctx.ask.yesNo(`Does an independent agent review ${what}?`, (ctx.doc.get(["gates", gate, "agent"]) ?? fallback.agent) !== false); ctx.doc.put(["gates", gate, "agent"], agent, fallback.agent); } + // Suggested yes where the file does not say: the schema's default (off) only keeps upgraded repos as they were. + const path = ["stages", "build", "acceptance_tests"]; + const own = ctx.doc.get(path); + const accept = await ctx.ask.yesNo("Write tests from Done when before each build?", typeof own === "boolean" ? own : true); + ctx.doc.put(path, accept, DEFAULTS.stages.build.acceptance_tests); }, async check() { diff --git a/templates/config.yaml b/templates/config.yaml index 49055824..13d787a9 100644 --- a/templates/config.yaml +++ b/templates/config.yaml @@ -35,10 +35,12 @@ gates: # Per-stage model, skills to load, and deterministic commands to run before and after. # A step of a stage can have its own model too, for example under a stage: # steps: { done-check: max } # a step's own model; see the step table in the README +# acceptance_tests: before each build, a session writes tests from the owner's Done when list +# (one more session per change); the build may not change them, and the final check reads them. stages: design: { model: strong, skills: [], before: [], after: [] } plan: { model: strong, skills: [], before: [], after: [] } - build: { model: default, skills: [], before: [], after: [], max_fix_loops: 3 } + build: { model: default, skills: [], before: [], after: [], max_fix_loops: 3, acceptance_tests: true } review: { model: strong, skills: [], before: [], after: [], max_rounds: 2 } verify: { model: default, skills: [], before: [], after: [] } diff --git a/templates/prompts/acceptance.md b/templates/prompts/acceptance.md new file mode 100644 index 00000000..8c3904f4 --- /dev/null +++ b/templates/prompts/acceptance.md @@ -0,0 +1,29 @@ +You write the acceptance tests for one change, in an isolated worktree on branch `intent/{{slug}}`, before anyone builds it. The code does not exist yet. Your tests hold the build to what the product owner asked for: the build may not change them. Do not commit: the runtime commits your work when you finish. + +The owner's Done when criteria: + +{{done_when}} + +Spec, for reference: + +{{spec}} + +Plan, for reference (what the build will change; you do not build it): + +{{plan}} + +What people told earlier sessions of this change (conversation.md); take it into account: + +{{conversation}} + +Rules: + +- Write only tests and test fixtures, in the project's own test style and test folders. Changing anything else is refused during this step. +- Write one or more tests for each criterion a test can check. Test the behaviour the owner described, through the interface the spec and plan name, not the details of how it will be built. +- If tests from an earlier build of this change are already on the branch, keep the ones that still fit and update the rest. +- Run the test command (`{{test_command}}`). Your new tests should fail now, because the change is not built yet; make sure they fail for that reason and not because the test itself is broken (a typo, a wrong import path the plan does not create). +- The shell commands you may run, with any arguments: {{commands}}. Others are refused, so do not try them. + +Return, in `tests`, one item per test you wrote: `criterion` (the owner's words), `file` (its path in the repository), and `test` (its name). Return, in `not_testable`, each criterion no test can check (for example how people feel about something, or something only seen in production), with `why` in a few plain words. Every criterion appears in one list or the other. + +Write a one-line `commit_message` for your tests. diff --git a/tests/unit/agents.test.ts b/tests/unit/agents.test.ts index 43725f1d..b5b6a07d 100644 --- a/tests/unit/agents.test.ts +++ b/tests/unit/agents.test.ts @@ -3,8 +3,9 @@ import { BUILTIN_AGENTS, CONTINUES_BUILD, inlineName, type Team, type TeamWay, C import { validateConfig } from "../../src/config"; test("the step sets", () => { - expect([...READ_ONLY_STEPS].some((k) => ["build", "reconcile", "fix", "revise"].includes(k))).toBe(false); + expect([...READ_ONLY_STEPS].some((k) => ["acceptance", "build", "reconcile", "fix", "revise"].includes(k))).toBe(false); expect(READ_ONLY_STEPS.has("review")).toBe(true); + // The acceptance step writes tests in a session of its own, before the build: never as the build's agent. expect([...CONTINUES_BUILD]).toEqual(["fix", "reconcile", "revise"]); expect([...CHAT_STEPS]).toEqual(["orchestrator", "write-intent"]); }); diff --git a/tests/unit/config.test.ts b/tests/unit/config.test.ts index a1ff2f3c..32d36c30 100644 --- a/tests/unit/config.test.ts +++ b/tests/unit/config.test.ts @@ -64,6 +64,17 @@ describe("loadConfig", () => { t.cleanup(); }); + test("stages.build.acceptance_tests is off unless set, so an upgraded repo is unchanged, and takes only true or false", async () => { + const t = tempDir(); + await writeConfig(t.path, "version: 1\ncommands:\n test: x\n"); + expect((await loadConfig(t.path)).stages.build.acceptance_tests).toBe(false); + await writeConfig(t.path, "version: 1\ncommands:\n test: x\nstages:\n build:\n acceptance_tests: true\n"); + expect((await loadConfig(t.path)).stages.build.acceptance_tests).toBe(true); + await writeConfig(t.path, "version: 1\ncommands:\n test: x\nstages:\n build:\n acceptance_tests: sometimes\n"); + await expect(loadConfig(t.path)).rejects.toThrow("stages.build.acceptance_tests: must be true or false"); + t.cleanup(); + }); + test("rejects a pull request as the done gate's surface, in plain words", async () => { const t = tempDir(); await writeConfig(t.path, "version: 1\ncommands:\n test: x\ngates:\n done:\n human: pr\n"); diff --git a/tests/unit/console-agent-settings.test.ts b/tests/unit/console-agent-settings.test.ts index 3ef291fe..46b5d329 100644 --- a/tests/unit/console-agent-settings.test.ts +++ b/tests/unit/console-agent-settings.test.ts @@ -110,7 +110,7 @@ describe("the agents view", () => { expect(view.steps[0]).toEqual({ key: "all", what: expect.any(String) as string, readOnly: true, chat: false, continuesBuild: false, own: { skills: ["tone"] }, fromAll: {}, fromStage: [] }); const row = (key: string) => view.steps.find((s) => s.key === key)!; - expect(view.steps.slice(1).map((s) => s.key)).toEqual(["intake", "design", "spec-check", "plan", "plan-challenge", "build", "reconcile", "fix", "review", "revise", "verify", "done-check", "lessons", "orchestrator", "write-intent"]); + expect(view.steps.slice(1).map((s) => s.key)).toEqual(["intake", "design", "spec-check", "plan", "plan-challenge", "acceptance", "build", "reconcile", "fix", "review", "revise", "verify", "done-check", "lessons", "orchestrator", "write-intent"]); expect(row("review")).toMatchObject({ readOnly: true, chat: false, continuesBuild: false, own: { agent: "lead", teams: ["reviewers"] }, fromAll: { skills: ["tone"] }, fromStage: ["house"] }); expect(row("fix")).toMatchObject({ readOnly: false, continuesBuild: true, own: {}, fromAll: { skills: ["tone"] }, fromStage: [] }); // Chat steps take nothing from all steps. diff --git a/tests/unit/envelopes.test.ts b/tests/unit/envelopes.test.ts index a36b6058..65008212 100644 --- a/tests/unit/envelopes.test.ts +++ b/tests/unit/envelopes.test.ts @@ -5,7 +5,7 @@ describe("envelopes", () => { test("every phase has a schema with the base fields", () => { const names: PhaseName[] = [ "intake", "design", "spec-check", "plan", "plan-challenge", "build", "fix", - "reconcile", "verify", "review", "revise", "done-check", "lessons", + "reconcile", "verify", "review", "revise", "done-check", "lessons", "acceptance", ]; for (const name of names) { const schema = Envelopes[name]; @@ -27,6 +27,18 @@ describe("envelopes", () => { expect(jsonSchemaFor("plan-challenge").properties).toHaveProperty("concerns"); }); + test("acceptance names each test by its criterion, file and name, and what it could not test; it never asks", () => { + const r = Envelopes.acceptance.parse({ + status: "success", summary: "one test, one criterion for a person", commit_message: "test: acceptance tests for counting words", + tests: [{ criterion: "Counting the words in \"hello world\" gives 2", file: "tests/count.test.ts", test: "counts two words" }], + not_testable: [{ criterion: "People find the new page easy to read", why: "needs a person" }], + }); + expect(r.tests[0]?.file).toBe("tests/count.test.ts"); + expect(r.not_testable[0]?.why).toBe("needs a person"); + expect(jsonSchemaFor("acceptance").required).toEqual(expect.arrayContaining(["tests", "not_testable", "commit_message"])); + expect(jsonSchemaFor("acceptance").properties).not.toHaveProperty("question"); + }); + test("intake accepts a priority and question; missing sections are the consistency check's, before intake", () => { const r = Envelopes.intake.parse({ status: "success", summary: "fine", diff --git a/tests/unit/models.test.ts b/tests/unit/models.test.ts index f1e3dc14..d581e1f9 100644 --- a/tests/unit/models.test.ts +++ b/tests/unit/models.test.ts @@ -10,14 +10,14 @@ test("every step's default is today's model", () => { const got = Object.fromEntries(STEP_KEYS.map((k) => [k, stepModel(DEFAULTS, k)])); expect(got).toEqual({ intake: "cheap", design: "strong", "spec-check": "strong", plan: "strong", "plan-challenge": "strong", - build: "default", reconcile: "default", fix: "default", review: "strong", revise: "default", + acceptance: "default", build: "default", reconcile: "default", fix: "default", review: "strong", revise: "default", verify: "default", "done-check": "strong", lessons: "cheap", orchestrator: "default", "write-intent": "strong", }); }); test("a step that follows a stage moves with it", () => { const cfg = validateConfig({ ...BASE, stages: { build: { model: "strong" } } }); - expect([stepModel(cfg, "reconcile"), stepModel(cfg, "fix"), stepModel(cfg, "revise")]).toEqual(["strong", "strong", "strong"]); + expect([stepModel(cfg, "acceptance"), stepModel(cfg, "reconcile"), stepModel(cfg, "fix"), stepModel(cfg, "revise")]).toEqual(["strong", "strong", "strong", "strong"]); }); test("a step's own setting wins", () => { @@ -44,6 +44,7 @@ test("the table lists every step key once, and each stage its own steps", () => expect(STEPS.map((s) => s.key)).toEqual([...STEP_KEYS]); expect(stepsOf("verify").map((s) => s.key)).toEqual(["verify", "done-check", "lessons"]); expect(stepsOf("chat").map((s) => s.key)).toEqual(["orchestrator", "write-intent"]); + expect(stepsOf("build").map((s) => s.key)).toEqual(["acceptance", "build", "reconcile", "fix"]); }); test("a group's model is its stage's, or chat's", () => { diff --git a/tests/unit/setup-gates.test.ts b/tests/unit/setup-gates.test.ts index 6b231838..1ddb12bf 100644 --- a/tests/unit/setup-gates.test.ts +++ b/tests/unit/setup-gates.test.ts @@ -12,8 +12,8 @@ describe("the gates section", () => { try { await Bun.write(`${repo.path}/package.json`, JSON.stringify({ scripts: { test: "bun test" } })); await init(repo.path); - // spec: a person, no agent; plan: as is; done: a person, as is. - const { shown } = await askSection(gates, repo.path, ["y", "n", "", "", "yes", ""]); + // spec: a person, no agent; plan: as is; done: a person, as is; acceptance tests: as is. + const { shown } = await askSection(gates, repo.path, ["y", "n", "", "", "yes", "", ""]); expect(shown).toContain("each step can wait for a person or go on by itself."); expect(shown).toContain("Should a person approve the spec (what will be built) before work goes on? (yes: they set its status in intent//intent.md; no: it goes on by itself)"); expect(shown).toContain("Should a person confirm the result after merging? (yes: they set its status in intent//intent.md; no: it is done by itself)"); @@ -28,11 +28,31 @@ describe("the gates section", () => { } finally { repo.cleanup(); } }); - test("on a config without gates, the defaults add nothing", async () => { + test("on a config without gates, the defaults add nothing but the suggested acceptance tests", async () => { const base = "version: 1\ncommands:\n test: echo ok\n"; const r = configRepo(base); try { - expect((await askSection(gates, r.root, "defaults")).text).toBe(base); + expect((await askSection(gates, r.root, "defaults")).text).toBe(`${base}stages:\n build:\n acceptance_tests: true\n`); + } finally { r.cleanup(); } + }); + + test("asks whether to write tests from Done when before each build, suggesting yes, and writes stages.build.acceptance_tests", async () => { + const repo = await tempGitRepo(); + try { + await Bun.write(`${repo.path}/package.json`, JSON.stringify({ scripts: { test: "bun test" } })); + await init(repo.path); + const { shown } = await askSection(gates, repo.path, ["", "", "", "", "", "", "n"]); + expect(shown).toContain("Write tests from Done when before each build? [Y/n]"); + expect((await loadConfig(repo.path)).stages.build.acceptance_tests).toBe(false); + expect(readFileSync(configPath(repo.path), "utf8")).toContain("acceptance_tests: false"); + await askSection(gates, repo.path, ["", "", "", "", "", "", "y"]); + expect((await loadConfig(repo.path)).stages.build.acceptance_tests).toBe(true); + } finally { repo.cleanup(); } + // A person's own no is the suggestion next time. + const r = configRepo("version: 1\ncommands:\n test: echo ok\nstages:\n build:\n acceptance_tests: false\n"); + try { + expect((await askSection(gates, r.root, ["", "", "", "", "", "", ""])).shown).toContain("Write tests from Done when before each build? [y/N]"); + expect(r.text()).toContain("acceptance_tests: false"); } finally { r.cleanup(); } }); }); diff --git a/tests/unit/templates.test.ts b/tests/unit/templates.test.ts index ad763e57..7fc915d5 100644 --- a/tests/unit/templates.test.ts +++ b/tests/unit/templates.test.ts @@ -62,6 +62,16 @@ describe("prompt templates", () => { expect(text).toMatch(/no test names, commands, file paths, or line numbers/); }); + test("the acceptance prompt has the Done when list, the spec, the plan, the test command and what people said; it writes only tests", async () => { + const text = await Bun.file(join(ROOT, "acceptance.md")).text(); + for (const v of ["done_when", "spec", "plan", "test_command", "conversation", "commands"]) expect({ v, used: text.includes(`{{${v}}}`) }).toEqual({ v, used: true }); + const rendered = renderPrompt(text, { done_when: "- add(1, 2) returns 3.", spec: "SPEC", plan: "PLAN", test_command: "bun test", conversation: "SAID", commands: "`bun test`" }); + expect(rendered).not.toMatch(/\{\{[a-z_]+\}\}/); + for (const part of ["- add(1, 2) returns 3.", "SPEC", "PLAN", "`bun test`", "SAID"]) expect(rendered).toContain(part); + expect(text).toMatch(/only tests and test fixtures/); + expect(text).toContain("`not_testable`"); + }); + test("every {{variable}} is a known prompt variable", async () => { const known = new Set(PROMPT_VARS); for (const name of Object.keys(Envelopes)) { @@ -99,6 +109,8 @@ describe("prompt templates", () => { const cfg = await loadConfig(repo.path); expect(cfg.claude.models).toEqual({ cheap: "haiku", default: "sonnet", strong: "opus", max: "fable" }); for (const s of STEPS) expect({ step: s.key, model: stepModel(cfg, s.key) }).not.toEqual({ step: s.key, model: "max" }); + // New repos write acceptance tests from Done when; the schema's default keeps upgraded repos as they were. + expect(cfg.stages.build.acceptance_tests).toBe(true); } finally { repo.cleanup(); } }); }); From d410617adb5d5e658f22109e058245cfb0af9ee9 Mon Sep 17 00:00:00 2001 From: John Flavan Date: Tue, 27 Oct 2026 21:50:03 -0500 Subject: [PATCH 04/21] feat(risk): gates.merge.risky settings and scoring a change's diff gates.merge.risky names patterns of files (Bun.Glob, repo paths with /), an optional size in lines, and who decides (pr or status). src/risk.ts scores a branch against main from where they parted, outside intent/, from git diff --numstat -z -M: binary files add no lines, a rename counts under whichever of its names a pattern matches. It also words the gate row's evidence and the owner's note. Co-Authored-By: Claude Opus 5.5 --- src/config.ts | 15 +++- src/risk.ts | 111 +++++++++++++++++++++++++++++ tests/unit/config.test.ts | 22 ++++++ tests/unit/init.test.ts | 3 +- tests/unit/risk.test.ts | 144 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 293 insertions(+), 2 deletions(-) create mode 100644 src/risk.ts create mode 100644 tests/unit/risk.test.ts diff --git a/src/config.ts b/src/config.ts index 74cd7c7a..97838d71 100644 --- a/src/config.ts +++ b/src/config.ts @@ -26,6 +26,19 @@ const humanGate = z.enum(["status", "pr", "none"]); const statusGate = z.enum(["status", "none"], { error: "must be status or none; a pull request cannot be used for this gate" }); const modelRef = z.enum(MODEL_REFS); +/** + * Changes a person must let in, whatever gates.merge.human says (see src/risk.ts): ones touching a + * file a pattern matches, or bigger than `lines`. Empty (the default), nothing is risky. + */ +const riskyGate = z.object({ + /** Patterns of repository paths, with `/`, as Bun.Glob reads them: `package.json` only at the top, `**\/package.json` anywhere. */ + paths: z.array(z.string().refine((p) => p.trim() !== "", "must be a pattern of files, like .github/**")).default([]), + /** Lines added plus removed, outside intent/, over which a change is risky. Unset: size never makes it so. */ + lines: z.number().int("must be a whole number").positive("must be more than 0").optional(), + /** Who decides a risky change. Without a remote, pr acts as status: there is no pull request to approve. */ + human: z.enum(["pr", "status"], { error: "must be pr or status; a risky change always waits for a person" }).default("pr"), +}).strict(); + type Steps = Partial>; /** Each step of a group may name its own model; unset, the step's built-in default applies (see STEPS). */ @@ -205,7 +218,7 @@ export const ConfigSchema = z.object({ gates: z.object({ spec: z.object({ human: statusGate.default("none"), agent: z.boolean().default(true) }).strict().prefault({}), plan: z.object({ human: statusGate.default("none"), agent: z.boolean().default(true) }).strict().prefault({}), - merge: z.object({ human: humanGate.default("none"), method: z.enum(["squash", "merge"]).default("squash") }).strict().prefault({}), + merge: z.object({ human: humanGate.default("none"), method: z.enum(["squash", "merge"]).default("squash"), risky: riskyGate.prefault({}) }).strict().prefault({}), // A pull request cannot confirm a result after the merge either. done: z.object({ human: statusGate.default("none"), agent: z.boolean().default(true) }).strict().prefault({}), }).strict().prefault({}), diff --git a/src/risk.ts b/src/risk.ts new file mode 100644 index 00000000..273450b3 --- /dev/null +++ b/src/risk.ts @@ -0,0 +1,111 @@ +import type { Config } from "./config"; +import type { Git } from "./git"; +import { andList } from "./text"; + +/* + * Which changes are risky: facts about the diff (its paths and size), never an agent's judgment, so + * the answer is predictable and testable. A risky change waits for a person before it merges, + * whatever gates.merge.human says (see passMergeGate in stages/merge.ts). + */ + +/** What the merge gate found about a change's diff against main. */ +export interface Risk { + risky: boolean; + /** The changed files a pattern in gates.merge.risky.paths matches, as git orders them. */ + files: string[]; + /** Lines added plus removed outside intent/; a binary file counts none. */ + lines: number; + /** `lines` is over gates.merge.risky.lines. */ + overLines: boolean; + /** gates.merge.risky.lines, when it is set. */ + limit?: number; +} + +/** True when the config names anything risky: patterns of files, or a size. */ +export function riskySet(cfg: Config): boolean { + const r = cfg.gates.merge.risky; + return r.paths.length > 0 || r.lines !== undefined; +} + +/** One file of `git diff --numstat -z`: its lines (0 for a binary file), and its paths (a rename's new one first). */ +interface Changed { lines: number; paths: string[] } + +/** + * The records of `git diff --numstat -z`: "added TAB removed TAB path NUL", or for a rename + * "added TAB removed TAB NUL old NUL new NUL". A binary file's counts are "-". + */ +function numstat(out: string): Changed[] { + const fields = out.split("\0"); + const changed: Changed[] = []; + for (let i = 0; i < fields.length; i++) { + const m = /^(-|\d+)\t(-|\d+)\t(.*)$/s.exec(fields[i] ?? ""); + if (!m) continue; + const lines = (m[1] === "-" ? 0 : Number(m[1])) + (m[2] === "-" ? 0 : Number(m[2])); + if (m[3]) { + changed.push({ lines, paths: [m[3]] }); + continue; + } + const from = fields[i + 1] ?? ""; + const to = fields[i + 2] ?? ""; + i += 2; + changed.push({ lines, paths: [to, from].filter(Boolean) }); + } + return changed; +} + +/** + * Scores the branch checked out in `wt` against `main` (from where the two parted, so only the + * branch's own changes count), outside intent/. A file counts when a pattern matches its path; a + * renamed one when a pattern matches its new path or its old one (moving a file out of a risky + * folder changes it as much as moving one in), listed under the name that matched. + */ +export async function scoreRisk(wt: Git, main: string, cfg: Config): Promise { + const { paths, lines: limit } = cfg.gates.merge.risky; + const globs = paths.map((p) => new Bun.Glob(p)); + // -z: names as they are (spaces, other alphabets); -M: a rename is one record, whatever diff.renames says. + const out = (await wt.run(["diff", "--numstat", "-z", "-M", `${main}...HEAD`])).out; + const files: string[] = []; + let lines = 0; + for (const c of numstat(out)) { + const outside = c.paths.filter((p) => !p.startsWith("intent/")); + if (!outside.length) continue; + lines += c.lines; + const hit = outside.find((p) => globs.some((g) => g.match(p))); + if (hit !== undefined) files.push(hit); + } + const overLines = limit !== undefined && lines > limit; + return { risky: files.length > 0 || overLines, files, lines, overLines, ...(limit !== undefined ? { limit } : {}) }; +} + +/** `names` joined in words, at most `max` of them, then how many more. */ +function someOf(names: readonly string[], max: number): string { + return andList(names.length > max ? [...names.slice(0, max), `${names.length - max} more`] : names); +} + +/** The merge gate's risk row: what made the change risky (up to 10 files), or "nothing risky". */ +export function riskEvidence(r: Risk): string { + if (!r.risky) return "nothing risky"; + const parts: string[] = []; + if (r.files.length) parts.push(`touches ${someOf(r.files, 10)}`); + if (r.overLines) parts.push(`${r.lines} lines changed (over ${r.limit ?? 0})`); + return parts.join("; "); +} + +/** + * The owner's note on a risky change waiting in merge-review: why (at most 3 files, so whoever + * approves knows what to look at), and how to let it in. `human` is who decides: pr when there is a + * pull request to approve, else status. + */ +export function riskNote(r: Risk, human: "status" | "pr"): string { + const why: string[] = []; + if (r.files.length) why.push(`This change touches files set to need a person before merging: ${someOf(r.files, 3)}.`); + if (r.overLines) { + why.push(r.files.length + ? `It also has ${r.lines} changed lines, more than the ${r.limit ?? 0} set.` + : `This change has ${r.lines} changed lines, more than the ${r.limit ?? 0} set to need a person before merging.`); + } + const how = human === "pr" + ? "Approve the pull request on GitHub, or set status to merge-approved." + : "Read review.md. To let this change in, change the status line to merge-approved."; + return [...why, how].join(" "); +} diff --git a/tests/unit/config.test.ts b/tests/unit/config.test.ts index a1ff2f3c..274705b8 100644 --- a/tests/unit/config.test.ts +++ b/tests/unit/config.test.ts @@ -108,6 +108,28 @@ describe("loadConfig", () => { expect(problems.map((p) => p.text)).toEqual(["commands.test is required: the single command that runs your tests"]); }); + test("gates.merge.risky is empty by default: nothing makes a change risky, and a pull request decides one that is", () => { + const cfg = validateConfig({ version: 1, commands: { test: "bun test" } }); + expect(cfg.gates.merge.risky).toEqual({ paths: [], human: "pr" }); + const set = validateConfig({ version: 1, commands: { test: "bun test" }, gates: { merge: { risky: { paths: [".github/**", "package.json"], lines: 1500, human: "status" } } } }); + expect(set.gates.merge.risky).toEqual({ paths: [".github/**", "package.json"], lines: 1500, human: "status" }); + }); + + test("gates.merge.risky refuses a blank pattern, a size that is not a positive whole number, nobody to decide, and keys it does not know", () => { + const problems = (risky: unknown) => { + try { validateConfig({ version: 1, commands: { test: "bun test" }, gates: { merge: { risky } } }); } catch (e) { return (e as ConfigError).problems.map((p) => ({ path: p.path.join("."), text: p.text, unknownKey: p.unknownKey })); } + return []; + }; + expect(problems({ paths: ["", " "] })).toEqual([ + { path: "gates.merge.risky.paths.0", text: "gates.merge.risky.paths.0: must be a pattern of files, like .github/**", unknownKey: false }, + { path: "gates.merge.risky.paths.1", text: "gates.merge.risky.paths.1: must be a pattern of files, like .github/**", unknownKey: false }, + ]); + expect(problems({ lines: 0 })).toEqual([{ path: "gates.merge.risky.lines", text: "gates.merge.risky.lines: must be more than 0", unknownKey: false }]); + expect(problems({ lines: 2.5 })).toEqual([{ path: "gates.merge.risky.lines", text: "gates.merge.risky.lines: must be a whole number", unknownKey: false }]); + expect(problems({ human: "none" })).toEqual([{ path: "gates.merge.risky.human", text: "gates.merge.risky.human: must be pr or status; a risky change always waits for a person", unknownKey: false }]); + expect(problems({ size: 3 })).toEqual([{ path: "gates.merge.risky", text: "gates.merge.risky: unknown key(s) size", unknownKey: true }]); + }); + test("an empty optional command reads as none, as it did before the blank-name rule", () => { const cfg = validateConfig({ version: 1, commands: { test: "bun test", lint: "", build: "", run: "", install: "" } }); expect([cfg.commands.lint, cfg.commands.build, cfg.commands.run, cfg.commands.install]).toEqual([undefined, undefined, undefined, undefined]); diff --git a/tests/unit/init.test.ts b/tests/unit/init.test.ts index 8c965d2d..7d5fe4c2 100644 --- a/tests/unit/init.test.ts +++ b/tests/unit/init.test.ts @@ -24,7 +24,8 @@ describe("init", () => { expect(cfg.commands.build).toBe("bun run build"); expect(cfg.commands.run).toBe("bun run start"); expect(cfg.gates.spec).toEqual({ human: "none", agent: true }); - expect(cfg.gates.merge).toEqual({ human: "none", method: "squash" }); + // The template names risky files only in a comment: nothing is risky until a person says so. + expect(cfg.gates.merge).toEqual({ human: "none", method: "squash", risky: { paths: [], human: "pr" } }); for (const f of ["intent/README.md", "intent/queue.md", "REVIEW.md", ".claude/skills/loopstra/SKILL.md", "loopstra/stamp.json"]) { expect(existsSync(join(t.path, f))).toBe(true); } diff --git a/tests/unit/risk.test.ts b/tests/unit/risk.test.ts new file mode 100644 index 00000000..12f0b170 --- /dev/null +++ b/tests/unit/risk.test.ts @@ -0,0 +1,144 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { mkdirSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { validateConfig, type Config } from "../../src/config"; +import { Git, NO_AUTO_MAINTENANCE } from "../../src/git"; +import { riskEvidence, riskNote, riskySet, scoreRisk, type Risk } from "../../src/risk"; +import { run, tempGitRepo } from "../helpers"; + +function config(risky: Record = {}): Config { + return validateConfig({ version: 1, commands: { test: "x" }, gates: { merge: { risky } } }); +} + +const cleanups: Array<() => void> = []; +afterEach(() => { for (const c of cleanups.splice(0)) c(); }); + +/** A repository on a branch off main, with `onMain` committed on main first and `onBranch` committed on the branch. */ +async function branchWith(onBranch: Record, onMain: Record = {}): Promise { + const repo = await tempGitRepo(); + cleanups.push(repo.cleanup); + const commit = async (files: Record, message: string) => { + for (const [path, content] of Object.entries(files)) { + mkdirSync(dirname(join(repo.path, path)), { recursive: true }); + writeFileSync(join(repo.path, path), content); + } + await run(["git", "add", "-A"], repo.path); + await run(["git", ...NO_AUTO_MAINTENANCE, "commit", "-q", "-m", message], repo.path); + }; + if (Object.keys(onMain).length) await commit(onMain, "main"); + await run(["git", "checkout", "-q", "-b", "intent/x"], repo.path); + await commit(onBranch, "branch"); + return new Git(repo.path); +} + +const lines = (n: number) => Array.from({ length: n }, (_, i) => `line ${i}`).join("\n") + "\n"; + +describe("riskySet", () => { + test("only paths or a size make anything risky", () => { + expect(riskySet(config())).toBe(false); + expect(riskySet(config({ human: "status" }))).toBe(false); + expect(riskySet(config({ paths: ["x"] }))).toBe(true); + expect(riskySet(config({ lines: 10 }))).toBe(true); + }); +}); + +describe("scoreRisk", () => { + test("with nothing set, no change is risky", async () => { + const wt = await branchWith({ ".github/workflows/ci.yml": "on: push\n" }); + expect(await scoreRisk(wt, "main", config())).toEqual({ risky: false, files: [], lines: 1, overLines: false }); + }); + + test("a pattern matches repository paths: .github/** anything under it, a bare name only at the top, **/ at any depth", async () => { + const wt = await branchWith({ + ".github/workflows/ci.yml": "on: push\n", + "package.json": "{}\n", + "web/package.json": "{}\n", + "db/migrations/1.sql": "create table t;\n", + "src/a.ts": "x\n", + }); + const r = await scoreRisk(wt, "main", config({ paths: [".github/**", "package.json", "**/migrations/**"] })); + expect(r.risky).toBe(true); + expect(r.files).toEqual([".github/workflows/ci.yml", "db/migrations/1.sql", "package.json"]); + expect(r.overLines).toBe(false); + }); + + test("records under intent/ never count, neither as files nor as lines", async () => { + const wt = await branchWith({ "intent/x/intent.md": lines(50), "src/a.ts": "x\n" }); + const r = await scoreRisk(wt, "main", config({ paths: ["**"], lines: 5 })); + expect(r).toEqual({ risky: true, files: ["src/a.ts"], lines: 1, overLines: false, limit: 5 }); + const only = await branchWith({ "intent/x/intent.md": lines(50) }); + expect(await scoreRisk(only, "main", config({ paths: ["**"], lines: 5 }))).toEqual({ risky: false, files: [], lines: 0, overLines: false, limit: 5 }); + }); + + test("lines added and deleted add up; over the size set is risky, at it is not", async () => { + const wt = await branchWith({ "src/a.ts": lines(4), "src/b.ts": "" }, { "src/b.ts": lines(2) }); + expect(await scoreRisk(wt, "main", config({ lines: 6 }))).toEqual({ risky: false, files: [], lines: 6, overLines: false, limit: 6 }); + expect(await scoreRisk(wt, "main", config({ lines: 5 }))).toEqual({ risky: true, files: [], lines: 6, overLines: true, limit: 5 }); + }); + + test("a binary file adds no lines, but its path still counts", async () => { + const wt = await branchWith({ "assets/logo.bin": new Uint8Array([0, 1, 2, 0, 255]) }); + expect(await scoreRisk(wt, "main", config({ paths: ["assets/**"], lines: 1 }))).toEqual({ risky: true, files: ["assets/logo.bin"], lines: 0, overLines: false, limit: 1 }); + }); + + test("a file renamed into a risky path counts under its new name; one renamed out counts under its old one", async () => { + const into = await branchWith({}, { "ci/build.yml": lines(30) }); + mkdirSync(join(into.cwd, ".github", "workflows"), { recursive: true }); + await into.run(["mv", "ci/build.yml", ".github/workflows/build.yml"]); + await into.run([...NO_AUTO_MAINTENANCE, "commit", "-q", "-m", "move"]); + expect(await scoreRisk(into, "main", config({ paths: [".github/**"] }))).toEqual({ risky: true, files: [".github/workflows/build.yml"], lines: 0, overLines: false }); + + const out = await branchWith({}, { ".github/workflows/build.yml": lines(30) }); + mkdirSync(join(out.cwd, "ci"), { recursive: true }); + await out.run(["mv", ".github/workflows/build.yml", "ci/build.yml"]); + await out.run([...NO_AUTO_MAINTENANCE, "commit", "-q", "-m", "move"]); + expect((await scoreRisk(out, "main", config({ paths: [".github/**"] }))).files).toEqual([".github/workflows/build.yml"]); + }); + + test("paths with spaces and letters outside ASCII match as they are", async () => { + const wt = await branchWith({ "docs/read me.md": "x\n", "docs/café.md": "y\n" }); + const r = await scoreRisk(wt, "main", config({ paths: ["docs/read me.md", "docs/caf*"] })); + expect(r.files).toEqual(["docs/café.md", "docs/read me.md"]); + }); + + test("only the branch's own changes count, not what main gained since it parted", async () => { + const wt = await branchWith({ "src/a.ts": "x\n" }); + await wt.run(["checkout", "-q", "main"]); + mkdirSync(join(wt.cwd, ".github"), { recursive: true }); + writeFileSync(join(wt.cwd, ".github", "ci.yml"), "on: push\n"); + await wt.run(["add", "-A"]); + await wt.run([...NO_AUTO_MAINTENANCE, "commit", "-q", "-m", "main moves"]); + await wt.run(["checkout", "-q", "intent/x"]); + expect((await scoreRisk(wt, "main", config({ paths: [".github/**"] }))).risky).toBe(false); + }); +}); + +describe("the words", () => { + const risk = (files: string[], over?: { lines: number; limit: number }): Risk => ({ + risky: files.length > 0 || !!over, files, lines: over?.lines ?? 3, overLines: !!over, ...(over ? { limit: over.limit } : {}), + }); + const many = Array.from({ length: 12 }, (_, i) => `f${i}.yml`); + + test("the gate row names up to 10 files, then how many more, and the size when it tripped", () => { + expect(riskEvidence(risk([]))).toBe("nothing risky"); + expect(riskEvidence(risk(["a", "b"]))).toBe("touches a and b"); + expect(riskEvidence(risk(many))).toBe(`touches ${many.slice(0, 10).join(", ")} and 2 more`); + expect(riskEvidence(risk([], { lines: 1612, limit: 1500 }))).toBe("1612 lines changed (over 1500)"); + expect(riskEvidence(risk(["a"], { lines: 1612, limit: 1500 }))).toBe("touches a; 1612 lines changed (over 1500)"); + }); + + test("the owner's note names up to 3 files and says who decides and how", () => { + expect(riskNote(risk([".github/workflows/ci.yml", "package.json"]), "pr")).toBe( + "This change touches files set to need a person before merging: .github/workflows/ci.yml and package.json. Approve the pull request on GitHub, or set status to merge-approved.", + ); + expect(riskNote(risk(["a", "b", "c", "d", "e"]), "status")).toBe( + "This change touches files set to need a person before merging: a, b, c and 2 more. Read review.md. To let this change in, change the status line to merge-approved.", + ); + expect(riskNote(risk([], { lines: 1612, limit: 1500 }), "status")).toBe( + "This change has 1612 changed lines, more than the 1500 set to need a person before merging. Read review.md. To let this change in, change the status line to merge-approved.", + ); + expect(riskNote(risk(["a"], { lines: 1612, limit: 1500 }), "pr")).toBe( + "This change touches files set to need a person before merging: a. It also has 1612 changed lines, more than the 1500 set. Approve the pull request on GitHub, or set status to merge-approved.", + ); + }); +}); From 728753fd345094350be9b4d719b9892692735b05 Mon Sep 17 00:00:00 2001 From: John Flavan Date: Tue, 27 Oct 2026 21:50:36 -0500 Subject: [PATCH 05/21] feat(acceptance): the hook keeps acceptance tests out of the build's reach The acceptance step may write only tests and test fixtures; build, fix, reconcile and revise may not edit the files in LOOPSTRA_ACCEPTANCE, and each refusal is logged for the block note. The fix rule is unchanged. Co-Authored-By: Claude Opus 5.5 --- templates/hooks/loopstra-protect-tests.ts | 35 ++++++++++++--- tests/unit/hook.test.ts | 55 +++++++++++++++++++++++ 2 files changed, 84 insertions(+), 6 deletions(-) diff --git a/templates/hooks/loopstra-protect-tests.ts b/templates/hooks/loopstra-protect-tests.ts index 888a1668..2d53f6e7 100644 --- a/templates/hooks/loopstra-protect-tests.ts +++ b/templates/hooks/loopstra-protect-tests.ts @@ -1,9 +1,15 @@ #!/usr/bin/env bun -// Claude Code PreToolUse hook. During a Loopstra fix phase, block edits to test files. +// Claude Code PreToolUse hook that keeps Loopstra's sessions off the tests they must not change: +// - the acceptance step writes only tests and test fixtures; +// - build, fix, reconcile and revise may not edit the acceptance tests ($LOOPSTRA_ACCEPTANCE, a JSON list); +// - fix may not edit test files, except tests the change added or changed on its branch. // Input: JSON on stdin with tool_name and tool_input. Exit 2 blocks the action and sends stderr to Claude. import { appendFileSync } from "node:fs"; const input = JSON.parse(await Bun.stdin.text()) as { tool_name?: string; tool_input?: { file_path?: string; path?: string } }; -if (process.env.LOOPSTRA_PHASE !== "fix") process.exit(0); +const phase = process.env.LOOPSTRA_PHASE ?? ""; +// The phases that change the code, and so may not change the tests written from Done when. +const EDITS_CODE = new Set(["build", "fix", "reconcile", "revise"]); +if (phase !== "acceptance" && !EDITS_CODE.has(phase)) process.exit(0); const slashes = (p: string) => p.replace(/\\/g, "/").replace(/\/+$/, ""); // The path is judged inside the project, so a project that itself sits under a tests/ folder is not all tests. // Claude Code sets CLAUDE_PROJECT_DIR; without it the hook runs in the project, so the working folder stands in. @@ -20,15 +26,32 @@ const isTest = /(^|\/)(tests?|__tests__|spec|[^/]+\.tests?(\.\w+)?)\//i.test(pat || /(^|\/)[^/]+\.\w*Tests?(\.\w+)?\//.test(path) || /\.(test|spec)s?\.[a-z0-9]+$|_test\.(go|py|rb|exs|cc|cpp|c|rs)$|_spec\.rb$|(^|\/)test_[^/]+\.py$/i.test(path) || /(^|\/)[A-Za-z0-9_]*[a-z0-9](Tests?\.(cs|fs|vb|java|kt|scala)|Tests\.swift)$/.test(path); -if (isTest && !changedOnBranch().some((f) => same(f) === same(path))) { - // The runtime names the file in the block note, so a person can make the edit instead of rebuilding. +if (phase === "acceptance") { + if (isTest) process.exit(0); + console.error(`Loopstra: the acceptance step writes only tests and test fixtures (${path}).`); + process.exit(2); +} +if (acceptanceTests().some((f) => same(f) === same(path))) refuse(`Loopstra: this is an acceptance test written from Done when; fix the code, not the test (${path}).`); +if (phase === "fix" && isTest && !changedOnBranch().some((f) => same(f) === same(path))) { + refuse(`Loopstra: test files are protected during a fix phase. Fix the code, not the test (${path}).`); +} +process.exit(0); + +/** Refuses the edit with `message`, and records the file, so the runtime names it in the block note and a person can make the edit instead of rebuilding. */ +function refuse(message: string): never { if (process.env.LOOPSTRA_PROTECTED_LOG) { try { appendFileSync(process.env.LOOPSTRA_PROTECTED_LOG, `${path}\n`); } catch { /* the refusal still stands */ } } - console.error(`Loopstra: test files are protected during a fix phase. Fix the code, not the test (${path}).`); + console.error(message); process.exit(2); } -process.exit(0); + +/** The acceptance test files ($LOOPSTRA_ACCEPTANCE), relative to the project with forward slashes. A list that cannot be read holds none. */ +function acceptanceTests(): string[] { + let list: unknown; + try { list = JSON.parse(process.env.LOOPSTRA_ACCEPTANCE ?? "[]"); } catch { return []; } + return Array.isArray(list) && list.every((f) => typeof f === "string") ? list.map((f: string) => slashes(f)) : []; +} /** * Files this change added or changed on its branch since it left main ($LOOPSTRA_BASE), relative to the project. diff --git a/tests/unit/hook.test.ts b/tests/unit/hook.test.ts index 3f7bbe36..1d46229c 100644 --- a/tests/unit/hook.test.ts +++ b/tests/unit/hook.test.ts @@ -25,6 +25,61 @@ describe("protect-tests hook", () => { expect(blocked.err).toContain("protected"); }); + test("the acceptance step may write only tests and test fixtures", async () => { + const acceptance = { LOOPSTRA_PHASE: "acceptance" }; + for (const p of ["/repo/tests/count.test.ts", "/repo/tests/fixtures/words.txt", "/repo/src/__tests__/count.ts", "C:\\repo\\tests\\count.test.ts"]) { + expect({ p, code: (await runHook({ tool_name: "Write", tool_input: { file_path: p } }, acceptance)).code }).toEqual({ p, code: 0 }); + } + for (const p of ["/repo/src/count.ts", "/repo/package.json", "/repo/docs/testing.md"]) { + expect({ p, code: (await runHook({ tool_name: "Edit", tool_input: { file_path: p } }, acceptance)).code }).toEqual({ p, code: 2 }); + } + const refused = await runHook({ tool_name: "Edit", tool_input: { file_path: "/repo/src/count.ts" } }, { ...acceptance, CLAUDE_PROJECT_DIR: "/repo" }); + expect(refused.err).toBe("Loopstra: the acceptance step writes only tests and test fixtures (src/count.ts).\n"); + }); + + test("build, fix, reconcile and revise may not edit an acceptance test; other phases and other files are left alone", async () => { + const t = tempDir(); + try { + const log = join(t.path, "protected-tests.txt"); + const accepted = JSON.stringify(["tests/count.test.ts", "tests/fixtures/words.txt"]); + const env = (phase: string) => ({ LOOPSTRA_PHASE: phase, LOOPSTRA_ACCEPTANCE: accepted, LOOPSTRA_PROTECTED_LOG: log, CLAUDE_PROJECT_DIR: "/repo" }); + for (const phase of ["build", "fix", "reconcile", "revise"]) { + const r = await runHook({ tool_name: "Edit", tool_input: { file_path: "/repo/tests/count.test.ts" } }, env(phase)); + expect({ phase, code: r.code }).toEqual({ phase, code: 2 }); + expect(r.err).toBe("Loopstra: this is an acceptance test written from Done when; fix the code, not the test (tests/count.test.ts).\n"); + } + expect(readFileSync(log, "utf8")).toBe("tests/count.test.ts\n".repeat(4)); + // A fixture the acceptance step wrote is held the same way. + expect((await runHook({ tool_name: "Write", tool_input: { file_path: "/repo/tests/fixtures/words.txt" } }, env("build"))).code).toBe(2); + // The build writes its own tests and code freely. + expect((await runHook({ tool_name: "Write", tool_input: { file_path: "/repo/tests/other.test.ts" } }, env("build"))).code).toBe(0); + expect((await runHook({ tool_name: "Edit", tool_input: { file_path: "/repo/src/count.ts" } }, env("revise"))).code).toBe(0); + // Phases that do not write code, and the acceptance step itself, are not held by the list. + expect((await runHook({ tool_name: "Edit", tool_input: { file_path: "/repo/tests/count.test.ts" } }, env("acceptance"))).code).toBe(0); + expect((await runHook({ tool_name: "Edit", tool_input: { file_path: "/repo/tests/count.test.ts" } }, env("review"))).code).toBe(0); + } finally { t.cleanup(); } + }); + + test("an acceptance test is matched through Windows backslashes, and its case only where the file system ignores case", async () => { + const accepted = { LOOPSTRA_PHASE: "build", LOOPSTRA_ACCEPTANCE: JSON.stringify(["tests/count.test.ts"]) }; + const backslashed = await runHook({ tool_name: "Edit", tool_input: { file_path: "C:\\repo\\tests\\count.test.ts" } }, { ...accepted, CLAUDE_PROJECT_DIR: "C:\\repo" }); + expect(backslashed.code).toBe(2); + expect(backslashed.err).toContain("(tests/count.test.ts)"); + // The list may be written with backslashes too. + const listed = { LOOPSTRA_PHASE: "fix", LOOPSTRA_ACCEPTANCE: JSON.stringify(["tests\\count.test.ts"]), CLAUDE_PROJECT_DIR: "/repo" }; + expect((await runHook({ tool_name: "Edit", tool_input: { file_path: "/repo/tests/count.test.ts" } }, listed)).code).toBe(2); + const caseless = process.platform === "win32" || process.platform === "darwin"; + const upper = await runHook({ tool_name: "Edit", tool_input: { file_path: "/repo/Tests/Count.test.ts" } }, { ...accepted, CLAUDE_PROJECT_DIR: "/repo" }); + expect(upper.code).toBe(caseless ? 2 : 0); + }); + + test("an unreadable acceptance list holds nothing back", async () => { + for (const list of ["not json", "{\"a\": 1}", "[1, 2]", ""]) { + const r = await runHook({ tool_name: "Edit", tool_input: { file_path: "/repo/tests/count.test.ts" } }, { LOOPSTRA_PHASE: "build", LOOPSTRA_ACCEPTANCE: list, CLAUDE_PROJECT_DIR: "/repo" }); + expect({ list, code: r.code }).toEqual({ list, code: 0 }); + } + }); + test("knows the test names of other languages: Go, pytest, .NET, Java", async () => { const fix = { LOOPSTRA_PHASE: "fix" }; for (const p of ["/repo/pkg/add_test.go", "/repo/app/calc_test.py", "/repo/src/App.Tests/CalcTests.cs", "/repo/Tests/X.cs", From 0a87cd63df58c410dff82f27d01f251a8f50e9b6 Mon Sep 17 00:00:00 2001 From: John Flavan Date: Tue, 27 Oct 2026 21:54:14 -0500 Subject: [PATCH 06/21] feat(risk): a change's merge_gate puts a person on its merge gate Frontmatter gains merge_gate (pr or status; blank reads as absent and gets no line). changeGates gives one change's gates: the config's, with merge_gate on the merge gate when the config has nobody there; it never takes a person off. waitsForPerson, isRunnable and offers apply it themselves, so the scheduler, queue.md, status, the console and chat all see a risky change as waiting for a person; apply's approve and send-back check, the attention note and plan-ahead's mayMergeFirst use it too. Co-Authored-By: Claude Opus 5.5 --- src/actions.ts | 15 +++++++++---- src/apply.ts | 4 ++-- src/attention.ts | 4 ++-- src/intents.ts | 30 ++++++++++++++++++++------ src/stages/plan.ts | 14 ++++++------ tests/unit/actions.test.ts | 3 +++ tests/unit/apply.test.ts | 14 ++++++++++++ tests/unit/intents-frontmatter.test.ts | 16 ++++++++++++++ tests/unit/intents-queue.test.ts | 23 +++++++++++++++++++- tests/unit/stages-design.test.ts | 7 +++--- tests/unit/status.test.ts | 16 ++++++++++++++ 11 files changed, 121 insertions(+), 25 deletions(-) diff --git a/src/actions.ts b/src/actions.ts index 8950babb..124b0937 100644 --- a/src/actions.ts +++ b/src/actions.ts @@ -1,5 +1,5 @@ import type { ActionKind, Offer } from "./console/types"; -import { isHeld, plainStatus, REVIEW_GATE, STATUSES, type Frontmatter, type HumanGates, type Priority, type Status } from "./intents"; +import { changeGates, isHeld, plainStatus, REVIEW_GATE, STATUSES, type Frontmatter, type HumanGates, type Priority, type Status } from "./intents"; import type { Trace } from "./trace"; /* @@ -38,7 +38,10 @@ export function isOpen(s: Status): boolean { return s !== "done" && s !== "closed"; } -/** True when a person sits on the gate of this review status (so Approve and Send back are theirs to give). */ +/** + * True when a person sits on the gate of this review status (so Approve and Send back are theirs to + * give). `human` is the change's own gates (see changeGates). + */ export function personReviews(s: Status, human: HumanGates): boolean { const gate = REVIEW_GATE[s]; return !!gate && human[gate] !== "none"; @@ -75,13 +78,17 @@ export function retryOptions(trace: Trace, slug: string, note: string): Status[] return retryTargets(note, fromBlock); } -/** What a change offers a person now. `retry`: its retry targets (see retryOptions); only a blocked change uses them. */ +/** + * What a change offers a person now: its own gates (see changeGates), so a risky change offers + * Approve and Send back though the config has nobody on the merge gate. `retry`: its retry targets + * (see retryOptions); only a blocked change uses them. + */ export function offers(fm: Frontmatter, human: HumanGates, retry: Status[]): Offer[] { const s = fm.status; const out: Offer[] = []; // An agent's question waits for a person: answering it is what the change needs first. if (isOpen(s) && fm.question.trim()) out.push({ kind: "answer", label: "Answer" }); - if (personReviews(s, human) && APPROVE_TO[s]) { + if (personReviews(s, changeGates(human, fm)) && APPROVE_TO[s]) { out.push({ kind: "approve", label: APPROVE_LABEL[s]!, to: APPROVE_TO[s] }); if (SEND_BACK_TO[s]) out.push({ kind: "send-back", label: "Send back", to: SEND_BACK_TO[s] }); } diff --git a/src/apply.ts b/src/apply.ts index 4cd25a0c..e6e5557e 100644 --- a/src/apply.ts +++ b/src/apply.ts @@ -5,7 +5,7 @@ import { APPROVE_TO, doneWords, isOpen, personReviews, questionChangedText, resu import { loadConfig, type Config } from "./config"; import { commitRecord, dropSentBack, landed, MainCheckoutMoved, statusPatch, writeSentBack } from "./context"; import { withEntry, type ConversationEntry } from "./conversation"; -import { changeExists, effectivePriority, humanGates, isHeld, parseIntentFile, readIntent, SLUG, writeIntent, type Frontmatter, type Intent, type Status } from "./intents"; +import { changeExists, changeGates, effectivePriority, humanGates, isHeld, parseIntentFile, readIntent, SLUG, writeIntent, type Frontmatter, type Intent, type Status } from "./intents"; import { anythingToApply, claimedRequests, claimRequest, CONSOLE, drainOld, finishRequest, pruneResults, requestsDir, saveApplying, waitingIds, type Outcome, type Request } from "./requests"; import type { RecordDraft } from "./record-draft"; import { Git, GitError } from "./git"; @@ -542,7 +542,7 @@ async function openChange(root: string, trace: Trace, req: FieldRequest | Answer /** Why a status request is not something the change can do from `expect`, or null when it is. */ function notAllowed(req: StatusRequest, fm: Frontmatter, cfg: Config, trace: Trace): string | null { - const human = humanGates(cfg); + const human = changeGates(humanGates(cfg), fm); switch (req.kind) { case "approve": case "send-back": if (!targetOf(req)) return `${req.slug} cannot be ${req.kind === "approve" ? "approved" : "sent back"} while it is ${req.expect}.`; diff --git a/src/attention.ts b/src/attention.ts index 993b9e8e..688c62de 100644 --- a/src/attention.ts +++ b/src/attention.ts @@ -2,7 +2,7 @@ import { loopDayNote, loopDaySpent } from "./budget"; import type { Config } from "./config"; import { ghProblem, GITHUB_SIGNAL, GITHUB_SILENT_MS } from "./github"; import { activePause, agoText, readLoopHold } from "./heartbeat"; -import { dependencyWait, hasQuestion, heldNote, humanGates, orderQueue, REVIEW_GATE, scanRepo, waitsForPerson, WATCHING_PR, type HumanGates, type Scan, type Status } from "./intents"; +import { changeGates, dependencyWait, hasQuestion, heldNote, humanGates, orderQueue, REVIEW_GATE, scanRepo, waitsForPerson, WATCHING_PR, type HumanGates, type Scan, type Status } from "./intents"; import { readQuestion, type Asked } from "./questions"; import { SYNC_SIGNAL } from "./remote"; import { humanNote, prWait } from "./stages/shared"; @@ -143,7 +143,7 @@ export async function attention(root: string, config: Config | { problem: string continue; } const gate = REVIEW_GATE[fm.status]!; - add("waiting", i.slug, title, fm.note || humanNote(GATE_DOC[gate].artifact, GATE_DOC[gate].approved, human[gate])); + add("waiting", i.slug, title, fm.note || humanNote(GATE_DOC[gate].artifact, GATE_DOC[gate].approved, changeGates(human, fm)[gate])); } return items; } diff --git a/src/intents.ts b/src/intents.ts index e9686c9e..542234df 100644 --- a/src/intents.ts +++ b/src/intents.ts @@ -48,6 +48,12 @@ export const Frontmatter = z.object({ hold: blankableString, /** "open" while an agent's question waits for a person (runtime-managed; see src/questions.ts). Blank reads as absent. */ question: blankableString, + /** + * Who lets this change merge when the config has nobody on the merge gate: written by the runtime + * when the change is risky (see src/risk.ts), removed when it is not. A person who deletes the line + * takes the change back to the config's gate. Blank reads as absent. + */ + merge_gate: z.preprocess((v) => (v === null || (typeof v === "string" && v.trim() === "") ? undefined : v), z.enum(["pr", "status"], { error: "must be pr or status, or the line removed" }).optional()), }).strict(); export type Frontmatter = z.infer; @@ -135,7 +141,7 @@ function parseSections(body: string): Record { } /** Runtime-managed fields that get no line while blank. */ -const BLANK_LEFT_OUT: ReadonlySet = new Set(["hold", "question"]); +const BLANK_LEFT_OUT: ReadonlySet = new Set(["hold", "question", "merge_gate"]); export function serializeIntentFile(file: IntentFile): string { const crlf = file.body.includes("\r\n"); @@ -434,20 +440,30 @@ export const REVIEW_GATE: Partial> = { }; /** - * Waiting for a person: blocked, a draft, or a review status whose gate has a person on it. (A - * merge-review with nobody on the gate waits for GitHub's checks, not for a person.) + * The gates of one change: the config's, with a person on the merge gate when the config has nobody + * there and the change's merge_gate names one (a risky change). merge_gate never takes a person off. + */ +export function changeGates(human: HumanGates, fm: Pick): HumanGates { + return human.merge === "none" && fm.merge_gate ? { ...human, merge: fm.merge_gate } : human; +} + +/** + * Waiting for a person: blocked, a draft, or a review status whose gate has a person on it, for this + * change (see changeGates). (A merge-review with nobody on the gate waits for GitHub's checks, not + * for a person.) */ export function waitsForPerson(intent: Intent, human: HumanGates): boolean { const s = intent.file.frontmatter.status; if (s === "blocked" || s === "draft") return true; const gate = REVIEW_GATE[s]; - return !!gate && human[gate] !== "none"; + return !!gate && changeGates(human, intent.file.frontmatter)[gate] !== "none"; } /** * Runnable: the runtime has something to do for this intent right now. - * A review status whose gate has a person on it is not runnable: it means the automated checks - * passed and a person is deciding; the scan picks up their status change (for example + * A review status whose gate has a person on it (for this change: see changeGates) is not + * runnable: it means the automated checks passed and a person is deciding; the scan picks up + * their status change (for example * merge-review → merge-approved, which is runnable). * With a remote, merge-review is always runnable: the merge step watches the pull request (merged * or closed on GitHub, its checks, and its approval when merge.human is pr). @@ -457,7 +473,7 @@ export function isRunnable(intent: Intent, human: HumanGates, hasRemote = false) if (s === "draft" || s === "blocked" || s === "done" || s === "closed") return false; if (s === "merge-review" && hasRemote) return true; const gate = REVIEW_GATE[s]; - if (gate && human[gate] !== "none") return false; + if (gate && changeGates(human, intent.file.frontmatter)[gate] !== "none") return false; return true; } diff --git a/src/stages/plan.ts b/src/stages/plan.ts index 681b8a52..abd616a3 100644 --- a/src/stages/plan.ts +++ b/src/stages/plan.ts @@ -2,7 +2,7 @@ import { roomBeside } from "../budget"; import { filesExistOrNew, parsePlanFiles } from "../checks"; import type { Config } from "../config"; import { clearMarker, writeMarker, type StepContext, type StepResult } from "../context"; -import { scanRepo, type Status } from "../intents"; +import { changeGates, humanGates, scanRepo, type Frontmatter } from "../intents"; import { agentPhase, phaseModel, phasePrompt, renderedThen, type AgentPhaseResult, type AgentPhaseSpec } from "../phases"; import { errorText } from "../shell"; import { notTheStepsFault } from "../stop"; @@ -74,11 +74,13 @@ function ruledOut(cfg: Config): string | null { } /** - * Another change in this status may merge into main before this change's plan step: its code would - * then not be the code the plan read. A pull request waiting for a person is not counted. + * Another change, at this frontmatter, may merge into main before this change's plan step: its code + * would then not be the code the plan read. A change waiting for a person in merge-review (the + * config's gate, or its own merge_gate: see changeGates) is not counted. */ -function mayMergeFirst(cfg: Config, status: Status): boolean { - return status === "reviewing" || status === "merge-approved" || (status === "merge-review" && cfg.gates.merge.human === "none"); +function mayMergeFirst(cfg: Config, fm: Frontmatter): boolean { + const s = fm.status; + return s === "reviewing" || s === "merge-approved" || (s === "merge-review" && changeGates(humanGates(cfg), fm).merge === "none"); } /** @@ -91,7 +93,7 @@ async function plansAhead(ctx: StepContext): Promise { const cfg = ctx.cfg; if (ruledOut(cfg) !== null || cfg.gates.spec.human !== "none" || !roomBeside(cfg, ctx.trace, ctx.slug, 2)) return false; const others = (await scanRepo(ctx.root)).intents.filter((i) => i.slug !== ctx.slug); - return !others.some((i) => mayMergeFirst(cfg, i.file.frontmatter.status)); + return !others.some((i) => mayMergeFirst(cfg, i.file.frontmatter)); } /** diff --git a/tests/unit/actions.test.ts b/tests/unit/actions.test.ts index 37c47d90..d4020a30 100644 --- a/tests/unit/actions.test.ts +++ b/tests/unit/actions.test.ts @@ -16,6 +16,9 @@ describe("what a change offers", () => { expect(kinds(offers(fm("plan-review"), nobody, []))).toEqual(["hold", "priority", "close"]); expect(offers(fm("verifying"), everyone, [])[0]).toEqual({ kind: "approve", label: "Confirm it is done", to: "done" }); expect(kinds(offers(fm("verifying"), everyone, []))).not.toContain("send-back"); + // A risky change's merge_gate puts a person on the merge gate the config left to nobody. + expect(kinds(offers(fm("merge-review", "merge_gate: pr\n"), nobody, []))).toEqual(["approve", "send-back", "hold", "priority", "close"]); + expect(kinds(offers(fm("merge-review"), nobody, []))).toEqual(["hold", "priority", "close"]); }); test("a draft offers start; a blocked change offers one retry per target; finished changes offer nothing", () => { diff --git a/tests/unit/apply.test.ts b/tests/unit/apply.test.ts index 76693a51..e6df76af 100644 --- a/tests/unit/apply.test.ts +++ b/tests/unit/apply.test.ts @@ -359,6 +359,20 @@ describe("applying requests", () => { } }); + test("a risky change's merge_gate puts a person on its merge gate: Approve the merge is theirs to give", async () => { + const risky = INTENT("merge-review", "x").replace("status: merge-review\n", "status: merge-review\nmerge_gate: pr\n"); + const r = await chatRepo({ intents: { a: risky, b: INTENT("merge-review", "y") } }); + try { + const q = submitRequest(r.root, { ...by, kind: "approve", slug: "a", expect: "merge-review" }); + const other = submitRequest(r.root, { ...by, kind: "approve", slug: "b", expect: "merge-review" }); + await apply(r.root); + expect(requestState(r.root, q.id)).toMatchObject({ ok: true, status: "merge-approved" }); + expect(requestState(r.root, other.id)).toMatchObject({ ok: false, text: "b cannot be approved now: nobody reviews that step, so the loop goes on by itself." }); + } finally { + r.cleanup(); + } + }); + test("retry only from blocked, even when an old block named the status", async () => { const r = await chatRepo({ intents: { a: INTENT("spec-review", "x") }, config: PEOPLE }); try { diff --git a/tests/unit/intents-frontmatter.test.ts b/tests/unit/intents-frontmatter.test.ts index 3445facd..2935a107 100644 --- a/tests/unit/intents-frontmatter.test.ts +++ b/tests/unit/intents-frontmatter.test.ts @@ -161,3 +161,19 @@ describe("question", () => { expect(text).toStartWith("---\nstatus: designing\n"); }); }); + +describe("merge_gate", () => { + test("pr or status is read; blank or absent reads as none and gets no line; anything else is a problem", () => { + const gated = parseIntentFile("---\nstatus: merge-review\nmerge_gate: pr\n---\n# Intent: x\n"); + expect(gated.frontmatter.merge_gate).toBe("pr"); + expect(serializeIntentFile(gated)).toContain("merge_gate: pr\n"); + expect(parseIntentFile("---\nstatus: merge-review\nmerge_gate: status\n---\n# Intent: x\n").frontmatter.merge_gate).toBe("status"); + for (const blank of ["merge_gate:\n", "merge_gate: \"\"\n", ""]) { + const none = parseIntentFile(`---\nstatus: merge-review\n${blank}---\n# Intent: x\n`); + expect(none.frontmatter.merge_gate).toBeUndefined(); + expect(serializeIntentFile(none)).not.toContain("merge_gate"); + } + expect(() => parseIntentFile("---\nstatus: merge-review\nmerge_gate: none\n---\n# Intent: x\n")).toThrow(FrontmatterProblem); + expect(() => parseIntentFile("---\nstatus: merge-review\nmerge_gate: none\n---\n# Intent: x\n")).toThrow("intent.md frontmatter problem at merge_gate: must be pr or status, or the line removed"); + }); +}); diff --git a/tests/unit/intents-queue.test.ts b/tests/unit/intents-queue.test.ts index 997b3a45..b4eb83eb 100644 --- a/tests/unit/intents-queue.test.ts +++ b/tests/unit/intents-queue.test.ts @@ -2,7 +2,7 @@ import { describe, expect, test } from "bun:test"; import { mkdirSync } from "node:fs"; import { join } from "node:path"; import { - checkConsistency, dependencyNote, isRunnable, orderQueue, parseIntentFile, plainStatus, readIntent, renderQueue, scanRepo, waitingOn, + changeGates, checkConsistency, dependencyNote, humanGates, isRunnable, waitsForPerson, orderQueue, parseIntentFile, plainStatus, readIntent, renderQueue, scanRepo, waitingOn, type Intent, } from "../../src/intents"; import { sameFileHolds } from "../../src/scheduler"; @@ -130,6 +130,27 @@ describe("scan and queue", () => { expect(isRunnable(with_("merged"), { spec: "none", plan: "none", merge: "none", done: "status" })).toBe(true); }); + test("a change's merge_gate puts a person on its merge gate when the config has nobody there, and never takes one off", () => { + const base = { slug: "x", dir: "", file: { frontmatter: { status: "merge-review", priority: "normal", author: "", opened: "", note: "", hold: "", question: "" }, title: "x", body: "", sections: {} }, artifacts: new Set() } as unknown as Intent; + const gated = (gate?: "pr" | "status"): Intent => ({ ...base, file: { ...base.file, frontmatter: { ...base.file.frontmatter, ...(gate ? { merge_gate: gate } : {}) } } }); + const nobody = humanGates(null); + const status = { ...nobody, merge: "status" } as const; + expect(changeGates(nobody, gated("pr").file.frontmatter)).toEqual({ ...nobody, merge: "pr" }); + expect(changeGates(nobody, gated().file.frontmatter)).toEqual(nobody); + // The config's own person stays: merge_gate only ever adds one. + expect(changeGates(status, gated("pr").file.frontmatter)).toEqual(status); + // Waits for a person, so it is not run without a remote, and is listed as waiting. + expect(waitsForPerson(gated("pr"), nobody)).toBe(true); + expect(waitsForPerson(gated("status"), nobody)).toBe(true); + expect(waitsForPerson(gated(), nobody)).toBe(false); + expect(isRunnable(gated("pr"), nobody)).toBe(false); + expect(isRunnable(gated("status"), nobody)).toBe(false); + expect(isRunnable(gated(), nobody)).toBe(true); + // With a remote, the merge step watches its pull request, as for any merge-review. + expect(isRunnable(gated("pr"), nobody, true)).toBe(true); + expect(renderQueue([gated("pr")], [], nobody)).toMatch(/## Needs a person[\s\S]*\| x \|/); + }); + test("writeIntent updates status and note and readIntent sees it", async () => { const t = tempDir(); await mk(t.path, "w", "status: accepted"); diff --git a/tests/unit/stages-design.test.ts b/tests/unit/stages-design.test.ts index e1539d03..fd1ebf2f 100644 --- a/tests/unit/stages-design.test.ts +++ b/tests/unit/stages-design.test.ts @@ -407,13 +407,14 @@ describe("plan ahead", () => { ["reviewing", "", false], ["merge-review", "", false], // no person on the merge gate: it merges once its checks pass ["merge-review", "gates:\n merge:\n human: status\n", true], // waiting for a person + ["merge-review", "risky", true], // a risky change: waiting for a person though nobody is on the config's gate ["building", "", true], ] as const) { - test(`with another change at ${status}${config ? " (a person on the merge gate)" : ""}, a plan is ${ahead ? "" : "not "}written ahead`, async () => { - const { repo, ctx, trace } = await setupRepo("accepted", { config }); + test(`with another change at ${status}${config === "risky" ? " (risky)" : config ? " (a person on the merge gate)" : ""}, a plan is ${ahead ? "" : "not "}written ahead`, async () => { + const { repo, ctx, trace } = await setupRepo("accepted", { config: config === "risky" ? "" : config }); const other = join(repo.path, "intent", "other-change"); mkdirSync(other, { recursive: true }); - const text = readFileSync(join(repo.path, "intent", "add-numbers", "intent.md"), "utf8").replace(/status: \S+/, `status: ${status}`); + const text = readFileSync(join(repo.path, "intent", "add-numbers", "intent.md"), "utf8").replace(/status: \S+/, `status: ${status}${config === "risky" ? "\nmerge_gate: pr" : ""}`); writeFileSync(join(other, "intent.md"), text); await runDesignStep(ctx); expect((await readIntent(repo.path, "add-numbers")).file.frontmatter.status).toBe("spec-approved"); diff --git a/tests/unit/status.test.ts b/tests/unit/status.test.ts index d17ab651..088f87af 100644 --- a/tests/unit/status.test.ts +++ b/tests/unit/status.test.ts @@ -103,6 +103,22 @@ describe("renderStatus", () => { } }); + test("a risky change waiting in merge-review needs a person though nobody is on the config's merge gate; its own note, else the gate's, says what to do", async () => { + const t = tempDir(); + try { + await config(t.path); + await intent(t.path, "risky", "status: merge-review\nmerge_gate: pr\nnote: \"This change touches files set to need a person before merging: package.json.\""); + await intent(t.path, "bare", "status: merge-review\nmerge_gate: status"); + await intent(t.path, "plain", "status: merge-review\nnote: Waiting for the automatic checks on GitHub."); + const text = await renderStatus(t.path, 1000); + expect(text).toContain("- Waiting for you (risky): This change touches files set to need a person before merging: package.json."); + expect(text).toContain("- Waiting for you (bare): Read review.md. When you are happy with it, change the status line to merge-approved."); + expect(text).not.toContain("Waiting for you (plain)"); + } finally { + t.cleanup(); + } + }); + test("a change whose pull request waits says why in place of the merge gate's note; one only an engineer can end needs attention whoever is on the gate", async () => { const t = tempDir(); try { From 44ebba893a9de727e95ccd5890806a0aa7805930 Mon Sep 17 00:00:00 2001 From: John Flavan Date: Tue, 27 Oct 2026 21:57:46 -0500 Subject: [PATCH 07/21] feat(risk): the merge gate decides whether a change is risky and records it passMergeGate scores the change in its worktree once the merge checks pass, on the code that will merge, and writes a merge/risk gate row (pass "nothing risky", or waiting with the files and size) when the config names anything risky. With nobody on the config's merge gate, a risky change gets merge_gate (gates.merge.risky.human) in the same record as its move to merge-review, with a note naming up to three files; a change that is not risky loses the field in that record (or in the record of its merge). Without a remote, merge_gate pr acts as status. mergeHuman gives a change's own merge gate; the pull request watcher, the no-remote merge-review wait and mergeRetry read it. setStatus takes frontmatter fields to set or remove in the same record. Co-Authored-By: Claude Opus 5.5 --- src/context.ts | 21 +++-- src/stages/merge.ts | 83 ++++++++++++++------ tests/integration/risk-gate.test.ts | 73 +++++++++++++++++ tests/unit/stages-review-merge.test.ts | 104 +++++++++++++++++++++++++ 4 files changed, 251 insertions(+), 30 deletions(-) create mode 100644 tests/integration/risk-gate.test.ts diff --git a/src/context.ts b/src/context.ts index f85e1ac2..e18d05bc 100644 --- a/src/context.ts +++ b/src/context.ts @@ -433,9 +433,9 @@ export async function commitForStep(ctx: StepContext, what: string, write: (draf * before writing, never over a status a person set while the step ran, nor into a change a person * deleted meanwhile (throws PersonChangedStatus instead). */ -export async function writeForStep(ctx: StepContext, patch: Partial, draft: RecordDraft): Promise { +export async function writeForStep(ctx: StepContext, patch: Partial, draft: RecordDraft, remove: Array = []): Promise { const expected = ctx.intent.file.frontmatter.status; - if (!(await writeIntent(ctx.intent, patch, { expectStatus: expected }, draft))) { + if (!(await writeIntent(ctx.intent, patch, { expectStatus: expected, remove }, draft))) { throwIfDeleted(ctx, draft); const found = parseIntentFile(draft.read(intentRel(ctx))!).frontmatter.status; throw new PersonChangedStatus(expected, found); @@ -464,14 +464,21 @@ export function statusPatch(from: Status, to: Status, note: string): Partial = {}): Promise { +export interface StatusFields { set?: Partial; remove?: Array } + +/** + * Writes the status and commits it (see commitRecord), with `fields` in the same record. `extra` + * goes into the traced status change, which is traced once the commit landed (a write tried again, + * or never committed, is not traced). + */ +export async function setStatus(ctx: StepContext, status: Status, note = "", extra: Record = {}, fields: StatusFields = {}): Promise { const from = ctx.intent.file.frontmatter.status; - const patch = statusPatch(from, status, note); + const patch = { ...fields.set, ...statusPatch(from, status, note) }; const landed = await commitForStep(ctx, `${from} → ${status}`, async (draft) => { - await writeForStep(ctx, patch, draft); + await writeForStep(ctx, patch, draft, fields.remove); }); if (!landed) return; ctx.trace.upsertIntent(ctx.slug, status, effectivePriority(ctx.intent.file.frontmatter)); diff --git a/src/stages/merge.ts b/src/stages/merge.ts index 0bfe6241..b03784af 100644 --- a/src/stages/merge.ts +++ b/src/stages/merge.ts @@ -1,12 +1,13 @@ import { existsSync } from "node:fs"; -import { block, clearMarker, onceMarker, PersonChangedStatus, readArtifact, readMarker, recordPersonEdits, setStatus, settleRecord, writeMarker, type Failure, type StepContext, type StepResult } from "../context"; +import { block, clearMarker, onceMarker, PersonChangedStatus, readArtifact, readMarker, recordPersonEdits, setStatus, settleRecord, writeMarker, type Failure, type StatusFields, type StepContext, type StepResult } from "../context"; import { evaluateGate, type Check } from "../gates"; import { Git, NO_AUTO_MAINTENANCE, passOn, statusPaths } from "../git"; import { GitHub, recordGitHub, type PrInfo } from "../github"; import { onCheckout, oneMerge, othersRunningCommands } from "../lanes"; -import type { Status } from "../intents"; +import { changeGates, humanGates, type Status } from "../intents"; import { codePhase } from "../phases"; import { deleteRemoteBranch, moveMain, pushBranch, shareMain, syncMain, type SyncOutcome } from "../remote"; +import { riskEvidence, riskNote, riskySet, scoreRisk, type Risk } from "../risk"; import { errorText } from "../shell"; import { markHealthPending } from "../signals"; import { endsStepQuietly, notTheStepsFault } from "../stop"; @@ -66,9 +67,22 @@ const FINISH_FAILING_MS = 30 * 60_000; /** A failure this long after the last one (the loop was off meanwhile) starts the count again. */ const FINISH_FAILING_GAP_MS = 10 * 60_000; -/** How a person asks the merge step to look again without a rebuild: the status that is runnable for this gate. */ +/** + * Who decides this change's merge: the config's merge gate, or the change's own merge_gate when the + * config has nobody there (see changeGates). Without a remote there is no pull request to approve, + * so a merge_gate of pr acts as status. + */ +function mergeHuman(ctx: StepContext, hasRemote: boolean): "none" | "status" | "pr" { + const gate = changeGates(humanGates(ctx.cfg), ctx.intent.file.frontmatter).merge; + return gate === "pr" && !hasRemote && ctx.cfg.gates.merge.human !== "pr" ? "status" : gate; +} + +/** + * How a person asks the merge step to look again without a rebuild: the status that is runnable for + * this change's gate. Only on the pull request's path, so a remote exists. + */ function mergeRetry(ctx: StepContext): Status { - return ctx.cfg.gates.merge.human === "status" ? "merge-approved" : "merge-review"; + return mergeHuman(ctx, true) === "status" ? "merge-approved" : "merge-review"; } /** What the merge gate's automated checks concluded. `changed`: they passed, but fixes were committed that no review has seen. */ @@ -140,27 +154,48 @@ async function checkMerge(ctx: StepContext): Promise { } /** - * The merge gate after its automated checks passed (the gate timing rule). With a remote: push the - * branch, open its pull request, and wait in merge-review (the merge step watches the pull request - * unless a person decides on the status line). Without one: no person → merge now; a person → - * merge-review with a note; a pull request → block plainly (there is nowhere to open one). + * The merge gate after its automated checks passed (the gate timing rule). First, whether the change + * is risky (see src/risk.ts), on the code that will merge: a risky change with nobody on the + * config's merge gate gets a person, gates.merge.risky.human, as its merge_gate, written in the same + * record as its move to merge-review, so nothing reads the status without it; a change that is not + * risky loses one an earlier pass wrote. With a remote: push the branch, open its pull request, and + * wait in merge-review (the merge step watches the pull request unless a person decides on the + * status line). Without one: no person → merge now; a person → merge-review with a note; a pull + * request set in the config → block plainly (there is nowhere to open one). */ async function passMergeGate(ctx: StepContext): Promise { - const human = ctx.cfg.gates.merge.human; - if ((await ctx.git.remoteName()) !== null) { + const hasRemote = (await ctx.git.remoteName()) !== null; + const risk = await riskOf(ctx); + const own = ctx.cfg.gates.merge.human === "none" && risk?.risky ? ctx.cfg.gates.merge.risky.human : undefined; + const fields = own ? { set: { merge_gate: own } } : { remove: ["merge_gate" as const] }; + // The note says why when the change's own gate brings in the person. + const human = own === "pr" && !hasRemote ? "status" : own ?? ctx.cfg.gates.merge.human; + const why = (r: Risk | null, otherwise: string) => (own && r ? riskNote(r, human === "pr" ? "pr" : "status") : otherwise); + if (hasRemote) { const opened = await openPullRequest(ctx, true); if (!opened.ok) return block(ctx, opened.note, opened); - await setStatus(ctx, "merge-review", human === "none" ? PR_CHECKS_NOTE : human === "pr" ? PR_APPROVE_NOTE : MERGE_WAIT_NOTE); + await setStatus(ctx, "merge-review", why(risk, human === "none" ? PR_CHECKS_NOTE : human === "pr" ? PR_APPROVE_NOTE : MERGE_WAIT_NOTE), {}, fields); return { ok: true }; } - if (human === "none") return mergeNow(ctx); + if (human === "none") return mergeNow(ctx, fields); if (human === "status") { - await setStatus(ctx, "merge-review", MERGE_WAIT_NOTE); + await setStatus(ctx, "merge-review", why(risk, MERGE_WAIT_NOTE), {}, fields); return { ok: true }; } return block(ctx, NO_REMOTE_PR_NOTE, { detail: "gates.merge.human is pr and the repository has no remote" }); } +/** + * Scores the change's branch in its worktree against main (see scoreRisk) and records it as the + * merge gate's risk row; null, with no row, when the config names nothing risky. + */ +async function riskOf(ctx: StepContext): Promise { + if (!riskySet(ctx.cfg)) return null; + const risk = await scoreRisk(new Git(ctx.worktreeDir), ctx.cfg.main_branch, ctx.cfg); + ctx.trace.gate(ctx.slug, "merge", "risk", risk.risky ? "waiting" : "pass", riskEvidence(risk)); + return risk; +} + /** * Shares main, pushes the branch, and makes sure it has an open pull request (title `: `, a body * naming the artifacts and the review summary): this build's own (see ownPr), or a new one. When a @@ -369,8 +404,10 @@ async function watchPullRequest(ctx: StepContext, gh: GitHub, unanswered: (error } const status = ctx.intent.file.frontmatter.status; + // Who decides this change: the config's merge gate, or its own merge_gate (a risky change). + const human = mergeHuman(ctx, true); // A person decides on the status line and has not yet: only a merge or close on GitHub counts. - if (ctx.cfg.gates.merge.human === "status" && status !== "merge-approved") return { ok: true, waiting: true }; + if (human === "status" && status !== "merge-approved") return { ok: true, waiting: true }; // A draft: GitHub refuses to merge it until a person marks it ready for review, so it is not // approved yet, whatever its checks say (traced once). if (pr.draft) { @@ -388,9 +425,9 @@ async function watchPullRequest(ctx: StepContext, gh: GitHub, unanswered: (error return block(ctx, PR_CHECKS_FAILED_NOTE, { detail: { pr: pr.number, url: pr.url }, retryFrom: mergeRetry(ctx) }); } // merge.human pr: approved on GitHub, or a person set merge-approved on the status line. - if (ctx.cfg.gates.merge.human === "pr" && !pr.approved && status !== "merge-approved") return { ok: true, waiting: true }; + if (human === "pr" && !pr.approved && status !== "merge-approved") return { ok: true, waiting: true }; ctx.trace.gate(ctx.slug, "merge", "pr-checks", "pass", checks === "none" ? `${where} (no checks reported)` : where); - if (ctx.cfg.gates.merge.human === "pr") ctx.trace.gate(ctx.slug, "merge", "pr-approved", "pass", pr.approved ? "approved on GitHub" : "merge-approved on the status line"); + if (human === "pr") ctx.trace.gate(ctx.slug, "merge", "pr-approved", "pass", pr.approved ? "approved on GitHub" : "merge-approved on the status line"); // At every speed.max_parallel_changes: the watcher runs beside the lanes even at one, so another // pull request may merge while this one waits, and it is then out of date. return mergeWhenUpToDate(ctx, gh, pr, waits); @@ -740,7 +777,7 @@ export async function runMergeStep(ctx: StepContext): Promise<StepResult> { } } if ((await ctx.git.remoteName()) !== null) return runRemoteMerge(ctx); - if (status === "merge-review" && ctx.cfg.gates.merge.human !== "none") return { ok: true }; + if (status === "merge-review" && mergeHuman(ctx, false) !== "none") return { ok: true }; // A person may have set merge-approved (or edited the change's other files) without committing: // record the change's own folder first, so their edit is part of main before the checks run. @@ -769,11 +806,11 @@ async function checkAgain(ctx: StepContext, onPass: () => Promise<StepResult>): * intent/ has unsaved changes, so a person's work there is never touched. Unsaved edits inside * intent/ (an owner's status lines) never block it; the merge takes only the branch. The look at * the checkout, the catch-up rebase, the marker and the move are one checkout-lock section, so no - * other change commits on main in between. + * other change commits on main in between. `fields` go into the record of the merge (see finishMerge). */ -async function mergeNow(ctx: StepContext): Promise<StepResult> { +async function mergeNow(ctx: StepContext, fields?: StatusFields): Promise<StepResult> { const landed = await onCheckout(ctx.root, () => mergeIntoMain(ctx)); - return landed === true ? finishMerge(ctx) : landed; + return landed === true ? finishMerge(ctx, fields) : landed; } /** @@ -858,11 +895,11 @@ async function alreadyMerged(ctx: StepContext): Promise<boolean> { return false; } -/** The change is on main: tidy up (best effort) and record it as merged. */ -async function finishMerge(ctx: StepContext): Promise<StepResult> { +/** The change is on main: tidy up (best effort) and record it as merged, with `fields` (see setStatus) in the same record. */ +async function finishMerge(ctx: StepContext, fields?: StatusFields): Promise<StepResult> { await cleanupChange(ctx); markHealthPending(ctx.root, ctx.slug); - await setStatus(ctx, "merged"); + await setStatus(ctx, "merged", "", {}, fields); clearMarker(ctx, MERGING); clearMarker(ctx, REVIEW_ROUND); clearMarker(ctx, REMOTE_AHEAD); diff --git a/tests/integration/risk-gate.test.ts b/tests/integration/risk-gate.test.ts new file mode 100644 index 00000000..e8f75472 --- /dev/null +++ b/tests/integration/risk-gate.test.ts @@ -0,0 +1,73 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { tick } from "../../src/scheduler"; +import { PR_CHECKS_NOTE } from "../../src/stages/shared"; +import { resetStop } from "../../src/stop"; +import { withEnv } from "../helpers"; +import { inTrace, intentOf, onRemoteMain, remoteSetup, SLUG } from "./remote-helpers"; + +afterEach(() => resetStop()); + +/** The build adds src/add.ts and tests/add.test.ts. */ +const risky = (paths: string, human = "") => `gates:\n merge:\n risky:\n paths: [${paths}]\n${human ? ` human: ${human}\n` : ""}`; + +describe("risky changes with a GitHub remote", () => { + test("nobody on the merge gate, a risky change: its pull request waits for an approval though its checks pass, then merges", async () => { + const s = await remoteSetup(risky('"src/**", "package.json"')); + await withEnv(s.env, async () => { + await tick(s.repo); // build + await tick(s.repo); // review, merge checks, pull request + const waiting = await intentOf(s.repo); + expect(waiting).toMatchObject({ status: "merge-review", merge_gate: "pr" }); + expect(waiting.note).toBe("This change touches files set to need a person before merging: src/add.ts. Approve the pull request on GitHub, or set status to merge-approved."); + expect(inTrace(s.repo, (t) => t.lastGate(SLUG, "merge", "risk"))).toMatchObject({ result: "waiting", evidence: "touches src/add.ts" }); + + // Its checks pass, but nobody approved: it does not merge, look after look. + await s.setPr({ checks: "pass" }); + await tick(s.repo); + await tick(s.repo); + expect((await intentOf(s.repo)).status).toBe("merge-review"); + expect((await s.pr()).merged).toBe(false); + expect(await onRemoteMain(s.remote, "src/add.ts")).toBe(false); + + await s.setPr({ reviewDecision: "APPROVED" }); + await tick(s.repo); + expect((await intentOf(s.repo)).status).toBe("merged"); + expect((await s.pr()).merged).toBe(true); + expect(await onRemoteMain(s.remote, "src/add.ts")).toBe(true); + expect(inTrace(s.repo, (t) => t.lastGate(SLUG, "merge", "pr-approved"))?.evidence).toBe("approved on GitHub"); + }); + s.cleanup(); + }, 180_000); + + test("a risky change decided on the status line waits for merge-approved, whatever GitHub says", async () => { + const s = await remoteSetup(risky('"src/**"', "status")); + await withEnv(s.env, async () => { + await tick(s.repo); + await tick(s.repo); + const waiting = await intentOf(s.repo); + expect(waiting).toMatchObject({ status: "merge-review", merge_gate: "status" }); + expect(waiting.note).toBe("This change touches files set to need a person before merging: src/add.ts. Read review.md. To let this change in, change the status line to merge-approved."); + await s.setPr({ checks: "pass", reviewDecision: "APPROVED" }); + await tick(s.repo); + expect((await intentOf(s.repo)).status).toBe("merge-review"); + expect((await s.pr()).merged).toBe(false); + }); + s.cleanup(); + }, 180_000); + + test("nobody on the merge gate, nothing risky touched: no merge_gate, a passing risk row, and it merges once its checks pass", async () => { + const s = await remoteSetup(risky('".github/**"')); + await withEnv(s.env, async () => { + await tick(s.repo); + await tick(s.repo); + const waiting = await intentOf(s.repo); + expect(waiting).toMatchObject({ status: "merge-review", note: PR_CHECKS_NOTE }); + expect(waiting.merge_gate).toBeUndefined(); + expect(inTrace(s.repo, (t) => t.lastGate(SLUG, "merge", "risk"))).toMatchObject({ result: "pass", evidence: "nothing risky" }); + await s.setPr({ checks: "pass" }); + await tick(s.repo); + expect((await intentOf(s.repo)).status).toBe("merged"); + }); + s.cleanup(); + }, 180_000); +}); diff --git a/tests/unit/stages-review-merge.test.ts b/tests/unit/stages-review-merge.test.ts index a824ca0b..58c8591c 100644 --- a/tests/unit/stages-review-merge.test.ts +++ b/tests/unit/stages-review-merge.test.ts @@ -106,6 +106,21 @@ async function landSquash(git: Git, branch: string, message: string): Promise<vo await git.run(["commit", "-q", "-m", message]); } +/** + * A copy of the built repository (nobody on the merge gate), with `config` appended to its config + * and committed, as an engineer's edit to it would be. + */ +async function builtWith(config: string) { + const s = await built(); + s.trace.close(); + await Bun.write(configPath(s.repo.path), `${await Bun.file(configPath(s.repo.path)).text()}${config}`); + await new Git(s.repo.path).commitAll("config"); + return { repo: s.repo, ...(await openRepo(s.repo.path)) }; +} + +/** The build adds src/add.ts and tests/add.test.ts: this pattern makes it risky. */ +const RISKY_SRC = "gates:\n merge:\n risky:\n paths: [\"src/**\"]\n"; + /** A person edits the status line by hand and commits it. */ async function personSets(ctx: StepContext, to: "merge-approved" | "done"): Promise<void> { await ctx.reload(); @@ -375,6 +390,95 @@ describe("merge", () => { trace.close(); repo.cleanup(); }); + test("a risky change without a remote waits in merge-review for merge-approved, though nobody is on the merge gate", async () => { + const { repo, ctx, trace } = await builtWith(RISKY_SRC); + await runMergeStep(ctx); + await ctx.reload(); + const fm = ctx.intent.file.frontmatter; + // Who decides is a pull request, which without a remote is the status line. + expect(fm).toMatchObject({ status: "merge-review", merge_gate: "pr" }); + expect(fm.note).toBe("This change touches files set to need a person before merging: src/add.ts. Read review.md. To let this change in, change the status line to merge-approved."); + expect(trace.lastGate("add-numbers", "merge", "risk")).toMatchObject({ result: "waiting", evidence: "touches src/add.ts" }); + expect(await onMain(repo.path, "src/add.ts")).toBe(false); + // The record that moved it to merge-review wrote the field: one commit, both lines. + const shown = (await new Git(repo.path).run(["show", "HEAD", "--", "intent/add-numbers/intent.md"])).out; + expect(shown).toContain("+status: merge-review"); + expect(shown).toContain("+merge_gate: pr"); + // Stepping the waiting status never merges. + await runMergeStep(ctx); + expect(await status(repo.path)).toBe("merge-review"); + expect(await onMain(repo.path, "src/add.ts")).toBe(false); + + await personSets(ctx, "merge-approved"); + await runMergeStep(ctx); + expect(await status(repo.path)).toBe("merged"); + expect(await onMain(repo.path, "src/add.ts")).toBe(true); + trace.close(); repo.cleanup(); + }); + + test("a change that touches nothing risky merges on its own, with a risk row that says so and no merge_gate", async () => { + const { repo, ctx, trace } = await builtWith("gates:\n merge:\n risky:\n paths: [\".github/**\"]\n lines: 500\n"); + await runMergeStep(ctx); + const i = await readIntent(repo.path, "add-numbers"); + expect(i.file.frontmatter.status).toBe("merged"); + expect(i.file.frontmatter.merge_gate).toBeUndefined(); + expect(gateRows(trace, "merge")).toEqual(["up-to-date:pass", "tests:pass", "findings:pass", "risk:pass"]); + expect(trace.lastGate("add-numbers", "merge", "risk")?.evidence).toBe("nothing risky"); + trace.close(); repo.cleanup(); + }); + + test("with nothing set to be risky, the merge gate writes no risk row", async () => { + const { repo, ctx, trace } = await built(); + await runMergeStep(ctx); + expect(gateRows(trace, "merge")).toEqual(["up-to-date:pass", "tests:pass", "findings:pass"]); + trace.close(); repo.cleanup(); + }); + + test("with a person on the config's merge gate, a risky change gets the risk row but no merge_gate", async () => { + const { repo, ctx, trace } = await builtWith("gates:\n merge:\n human: status\n risky:\n paths: [\"src/**\"]\n human: pr\n"); + await runMergeStep(ctx); + const i = await readIntent(repo.path, "add-numbers"); + expect(i.file.frontmatter.status).toBe("merge-review"); + expect(i.file.frontmatter.merge_gate).toBeUndefined(); + expect(i.file.frontmatter.note).toBe(MERGE_WAIT_NOTE); + expect(trace.lastGate("add-numbers", "merge", "risk")?.result).toBe("waiting"); + trace.close(); repo.cleanup(); + }); + + test("a merge_gate a change no longer earns is removed at its next pass of the merge gate, and it merges", async () => { + const { repo, ctx, trace } = await builtWith(RISKY_SRC); + await runMergeStep(ctx); + expect((await readIntent(repo.path, "add-numbers")).file.frontmatter.merge_gate).toBe("pr"); + // Built again without the risky files, as far as the gate can tell: the pattern no longer matches. + trace.close(); + const config = (await Bun.file(configPath(repo.path)).text()).replace('paths: ["src/**"]', 'paths: ["docs/**"]'); + await Bun.write(configPath(repo.path), config); + await new Git(repo.path).commitAll("config"); + const again = await openRepo(repo.path); + await writeIntentOnDisk(again.ctx.intent, { status: "reviewing", note: "" }); + await again.ctx.git.commitPaths(["intent/add-numbers"], "person: reviewing"); + await again.ctx.reload(); + await runMergeStep(again.ctx); + const i = await readIntent(repo.path, "add-numbers"); + expect(i.file.frontmatter.status).toBe("merged"); + expect(i.file.frontmatter.merge_gate).toBeUndefined(); + expect(await Bun.file(join(repo.path, "intent", "add-numbers", "intent.md")).text()).not.toContain("merge_gate"); + again.trace.close(); repo.cleanup(); + }); + + test("a person who deletes the merge_gate line takes the change back to the config's gate: the merge step merges it", async () => { + const { repo, ctx, trace } = await builtWith(RISKY_SRC); + await runMergeStep(ctx); + await ctx.reload(); + await writeIntentOnDisk(ctx.intent, {}, { remove: ["merge_gate"] }); + await ctx.git.commitPaths(["intent/add-numbers"], "person: no merge_gate"); + await ctx.reload(); + expect(ctx.intent.file.frontmatter.merge_gate).toBeUndefined(); + await runMergeStep(ctx); + expect(await status(repo.path)).toBe("merged"); + trace.close(); repo.cleanup(); + }); + test("a person sets merge-approved without committing: the merge proceeds, and main gets one commit with the branch's tree", async () => { const { repo, ctx, trace } = await reviewed(PERSON_MERGES); const git = new Git(repo.path); From f9494887d63252a5b4cc05f39eaf2ac95da2dd7a Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 21:58:38 -0500 Subject: [PATCH 08/21] feat(retro): loopstra retro: what shipped, where changes blocked, gates, spending, waits, and suggestions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- src/cli.ts | 15 +- src/commands/args.ts | 26 ++- src/commands/retro.ts | 31 +++ src/judges.ts | 6 +- src/phases.ts | 5 +- src/retro.ts | 432 ++++++++++++++++++++++++++++++++++++++ src/stages/build.ts | 13 +- src/stages/merge.ts | 8 +- src/stages/review.ts | 5 +- src/stages/verify.ts | 2 +- tests/unit/cli.test.ts | 51 ++++- tests/unit/judges.test.ts | 8 + tests/unit/retro.test.ts | 408 +++++++++++++++++++++++++++++++++++ 13 files changed, 981 insertions(+), 29 deletions(-) create mode 100644 src/commands/retro.ts create mode 100644 src/retro.ts create mode 100644 tests/unit/retro.test.ts diff --git a/src/cli.ts b/src/cli.ts index 52f6d315..88150708 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -21,6 +21,7 @@ const HELP = `loopstra <command> status show intents, phases, and blocks tail stream events (tail <slug> for one change) judges how often the cheaper judges agreed with the strong ones + retro how the loop did lately, and what to change (--days <n>, default 7) ui the console: live agents, inbox, changes and chat (--port <n>, default 4646; --no-chat) chat talk to the orchestrator: updates, and new changes written up as intents (--no-terminal to run only the Slack/Discord bots) @@ -33,12 +34,13 @@ const HELP = `loopstra <command> `; /** The words each command takes (see parseArgs); the usage line is said when they do not fit. */ -const ARGS: Record<"init" | "status" | "start" | "tail" | "judges" | "ui" | "chat" | "apply-lessons" | "upgrade" | "customize", ArgSpec> = { +const ARGS: Record<"init" | "status" | "start" | "tail" | "judges" | "retro" | "ui" | "chat" | "apply-lessons" | "upgrade" | "customize", ArgSpec> = { init: { usage: "Usage: loopstra init" }, status: { usage: "Usage: loopstra status" }, start: { usage: "Usage: loopstra start [--once]", flags: ["--once"] }, tail: { usage: "Usage: loopstra tail [slug]", names: 1 }, judges: { usage: "Usage: loopstra judges" }, + retro: { usage: "Usage: loopstra retro [--days <number>]", days: true }, ui: { usage: "Usage: loopstra ui [--port <number>] [--no-chat]", flags: ["--no-chat"], port: true }, chat: { usage: "Usage: loopstra chat [--no-terminal]", flags: ["--no-terminal"] }, "apply-lessons": { usage: "Usage: loopstra apply-lessons <slug>", names: 1 }, @@ -53,7 +55,7 @@ function args(command: keyof typeof ARGS): Args | null { return parsed; } -/** status, tail, judges, ui, chat, upgrade, and customize read a set-up repo; elsewhere they say so and create nothing. */ +/** status, tail, judges, retro, ui, chat, upgrade, and customize read a set-up repo; elsewhere they say so and create nothing. */ function setUp(): boolean { if (existsSync(configPath(root))) return true; console.error(NOT_SET_UP); @@ -164,6 +166,15 @@ async function main(): Promise<number> { return 0; } catch (e) { console.error(errorText(e)); return 1; } } + case "retro": { + const a = args("retro"); + if (!a || !setUp()) return 1; + const { DEFAULT_DAYS, renderRetro } = await import("./commands/retro"); + try { + process.stdout.write(await renderRetro(root, a.days ?? DEFAULT_DAYS)); + return 0; + } catch (e) { console.error(errorText(e)); return 1; } + } case "ui": { const a = args("ui"); if (!a || !setUp() || olderSetup("run loopstra ui again")) return 1; diff --git a/src/commands/args.ts b/src/commands/args.ts index d3ab5040..9778c83b 100644 --- a/src/commands/args.ts +++ b/src/commands/args.ts @@ -6,6 +6,8 @@ export interface ArgSpec { flags?: readonly string[]; /** True when it takes `--port <n>` or `--port=<n>`. */ port?: boolean; + /** True when it takes `--days <n>` or `--days=<n>`. */ + days?: boolean; /** How many plain words (a change name, a prompt name) it takes; none by default. */ names?: number; } @@ -16,34 +18,48 @@ export interface Args { flags: Set<string>; /** The port given, or null for the command's default. */ port: number | null; + /** The days given, or null for the command's default. */ + days: number | null; /** The plain words, in order. */ names: string[]; } -const PORT = /^\d+$/; +const WHOLE = /^\d+$/; const MAX_PORT = 65535; +/** The most days a report may look back: ten years. */ +const MAX_DAYS = 3650; /** * Sorts a command's words by `spec`, or gives null when they do not fit, for the caller to print * the usage line: a word starting with `-` that the command does not take (`--onec`, `-p`), a flag * given twice, a port that is not a whole number from 0 to 65535 (`--port ""`, `--port=5000x`, - * `--port` with nothing after it), or more plain words than it takes. Nothing is guessed from a - * word that is nearly a flag: the loop runs unattended, and `start --onec` must not run forever. + * `--port` with nothing after it), days that are not a whole number from 1 to 3650, or more plain + * words than it takes. Nothing is guessed from a word that is nearly a flag: the loop runs + * unattended, and `start --onec` must not run forever. */ export function parseArgs(args: readonly string[], spec: ArgSpec): Args | null { - const out: Args = { flags: new Set(), port: null, names: [] }; + const out: Args = { flags: new Set(), port: null, days: null, names: [] }; const takes = new Set(spec.flags ?? []); const port = (text: string | undefined): boolean => { - if (!spec.port || out.port !== null || text === undefined || !PORT.test(text) || Number(text) > MAX_PORT) return false; + if (!spec.port || out.port !== null || text === undefined || !WHOLE.test(text) || Number(text) > MAX_PORT) return false; out.port = Number(text); return true; }; + const days = (text: string | undefined): boolean => { + if (!spec.days || out.days !== null || text === undefined || !WHOLE.test(text) || Number(text) < 1 || Number(text) > MAX_DAYS) return false; + out.days = Number(text); + return true; + }; for (let i = 0; i < args.length; i++) { const word = args[i]!; if (word === "--port") { if (!port(args[++i])) return null; } else if (word.startsWith("--port=")) { if (!port(word.slice("--port=".length))) return null; + } else if (word === "--days") { + if (!days(args[++i])) return null; + } else if (word.startsWith("--days=")) { + if (!days(word.slice("--days=".length))) return null; } else if (word.startsWith("-")) { if (!takes.has(word) || out.flags.has(word)) return null; out.flags.add(word); diff --git a/src/commands/retro.ts b/src/commands/retro.ts new file mode 100644 index 00000000..d632dd0b --- /dev/null +++ b/src/commands/retro.ts @@ -0,0 +1,31 @@ +import { loadConfig } from "../config"; +import { buildRetro, retroText, windowStart } from "../retro"; +import { Trace } from "../trace"; +import { judgesView } from "./judges"; + +/** How many days `loopstra retro` looks back without --days. */ +export const DEFAULT_DAYS = 7; + +/** + * `loopstra retro`: how the loop did over the last `days` days, and what to change. Read-only. Every + * status move is read, not only the window's: a change merged in the window may have been accepted before it. + */ +export async function renderRetro(root: string, days: number): Promise<string> { + const cfg = await loadConfig(root); + const trace = Trace.open(root); + try { + const now = new Date(); + const since = windowStart(now, days); + return retroText(buildRetro({ + days, now, cfg, + moves: trace.statusChangesSince(new Date(0).toISOString()), + gates: trace.gatesSince(since), + phases: trace.phasesSince(since), + questions: trace.eventsOfTypeSince("question", since), + requests: trace.eventsOfTypeSince("request", since), + judges: judgesView(root, cfg, trace), + })); + } finally { + trace.close(); + } +} diff --git a/src/judges.ts b/src/judges.ts index e7352dea..9de43384 100644 --- a/src/judges.ts +++ b/src/judges.ts @@ -152,6 +152,8 @@ export interface JudgeReportRow { firstWouldCost: number | null; /** Whether it goes first, in words; null when the setting is off or it cannot be compared. */ goesFirst: string | null; + /** Its record has earned going first (see firstStanding); false when it cannot be compared. */ + earned: boolean; } export interface JudgeReport { @@ -179,7 +181,7 @@ function goesFirstText(setting: JudgeReport["setting"], s: FirstStanding): strin } /** One judge's counts over its rows with the current key. */ -function reportRow(key: JudgeKey | null, rows: readonly JudgeRow[]): { row: Omit<JudgeReportRow, "judge" | "why" | "goesFirst">; standing: FirstStanding } { +function reportRow(key: JudgeKey | null, rows: readonly JudgeRow[]): { row: Omit<JudgeReportRow, "judge" | "why" | "goesFirst" | "earned">; standing: FirstStanding } { const mine = key ? rows.filter((r) => matches(r, key)) : []; const shadow = mine.filter((r) => r.how === "shadow"); const compared = shadow.filter((r) => r.cheap !== "failed" && r.strong !== "failed"); @@ -227,7 +229,7 @@ export function judgeReport(cfg: Config, rows: readonly JudgeRow[], templates: P judges: JUDGES.map((judge) => { const { row, standing } = reportRow(keyOf(judge), rows); const why = notComparable(cfg, judge); - return { judge, why, ...row, goesFirst: why ? null : goesFirstText(setting, standing) }; + return { judge, why, ...row, goesFirst: why ? null : goesFirstText(setting, standing), earned: !why && standing.earned }; }), notCounted: rows.filter((r) => !counted(r)).length, falsePasses: rows.filter((r) => counted(r) && r.how === "shadow" && r.cheap === "pass" && r.strong === "reject").reverse(), diff --git a/src/phases.ts b/src/phases.ts index 41ddcc4e..8cbddf91 100644 --- a/src/phases.ts +++ b/src/phases.ts @@ -162,11 +162,14 @@ export function disallowedFor(set: ToolSet): string[] { /** Failures that block the change; an unavailable assistant pauses the loop instead (see unavailable). */ type BlockingReason = Exclude<FailureReason, "environment" | "not-started">; +/** The owner's note when a session hit its spending limit. */ +export const SESSION_LIMIT_NOTE = "This step hit its spending limit (claude.max_budget_usd). An engineer can raise or remove it with `loopstra setup budgets`."; + /** One plain sentence per failure reason, for the owner. The raw detail goes to the trace. */ export function ownerNote(reason: BlockingReason): string { switch (reason) { case "timeout": return `${TIMEOUT_NOTE} An engineer can allow longer with claude.timeout_minutes in loopstra/config.yaml.`; - case "budget": return "This step hit its spending limit (claude.max_budget_usd). An engineer can raise or remove it with `loopstra setup budgets`."; + case "budget": return SESSION_LIMIT_NOTE; case "crash": return CRASH_NOTE; case "no-session": return "The assistant could not pick up its earlier work."; case "model": return "Claude Code did not accept this step's model. Change it in Settings or `loopstra setup models`, then retry."; diff --git a/src/retro.ts b/src/retro.ts new file mode 100644 index 00000000..2445aba1 --- /dev/null +++ b/src/retro.ts @@ -0,0 +1,432 @@ +import { z } from "zod"; +import { CHAT_SLUG } from "./budget"; +import type { Config } from "./config"; +import { CRASH_NOTE, TIMEOUT_NOTE } from "./context"; +import { phaseKind } from "./console/steerable"; +import { humanGates, REVIEW_GATE, type Status } from "./intents"; +import { JUDGES, type JudgeName, type JudgeReport } from "./judges"; +import { SESSION_LIMIT_NOTE } from "./phases"; +import { TOO_MANY_NOTE } from "./questions"; +import { CHECKS_KEPT_TIMING_OUT_NOTE, TESTS_KEPT_FAILING_NOTE, VERIFY_STILL_FAILED_NOTE } from "./stages/build"; +import { PR_CHECKS_FAILED_NOTE, PR_CLOSED_NOTE, REVIEWS_USED_UP, UPDATE_FAILED_NOTE } from "./stages/merge"; +import { REVIEW_STILL_FOUND_NOTE } from "./stages/review"; +import { PR_CHECKS_NOTE } from "./stages/shared"; +import { UNMET_NOTE } from "./stages/verify"; +import { andList } from "./text"; +import type { EventRow, GateRow, PhaseRow, StatusMove } from "./trace"; + +/* + * `loopstra retro`: how the loop did over the last days, and what to change. Pure: it reads rows the + * trace gave (see src/commands/retro.ts), the config and the judges' report, and writes nothing. + */ + +/** Why a change blocked, as the note it blocked with tells. */ +export type BlockKind = "timeout" | "crash" | "tests" | "review" | "verify" | "budget" | "conflicts" | "pr-checks" | "pr-closed" | "unmet" | "questions" | "other"; + +/** + * The notes each kind of block is written from, in the order they are looked for. A failed session's + * note (a timeout, a crash, the spending limit, too many questions) can follow a stage's own words ("The + * finished change could not be checked. The assistant took too long..."), so those come first. + */ +const BLOCK_NOTES: ReadonlyArray<readonly [BlockKind, readonly string[]]> = [ + ["timeout", [TIMEOUT_NOTE]], + ["crash", [CRASH_NOTE]], + ["budget", [SESSION_LIMIT_NOTE]], + ["questions", [TOO_MANY_NOTE]], + ["tests", [TESTS_KEPT_FAILING_NOTE, CHECKS_KEPT_TIMING_OUT_NOTE]], + ["review", [REVIEW_STILL_FOUND_NOTE, REVIEWS_USED_UP]], + ["verify", [VERIFY_STILL_FAILED_NOTE]], + ["conflicts", [UPDATE_FAILED_NOTE]], + ["pr-checks", [PR_CHECKS_FAILED_NOTE]], + ["pr-closed", [PR_CLOSED_NOTE]], + ["unmet", [UNMET_NOTE]], +]; + +/** Every kind, in the order the report lists kinds with as many blocks. */ +const KINDS: readonly BlockKind[] = [...BLOCK_NOTES.map(([k]) => k), "other"]; + +/** Each kind in words, for the report. */ +const KIND_WORDS: Record<BlockKind, string> = { + timeout: "Timed out", crash: "The assistant crashed", tests: "Tests kept failing", review: "The reviewer kept finding problems", + verify: "The verifier found it did not work", budget: "Hit the spending limit", conflicts: "Merge conflicts", + "pr-checks": "Checks on GitHub failed", "pr-closed": "The pull request was closed", unmet: "A Done when criterion was not met", + questions: "Asked too many questions", other: "Other", +}; + +/** The kind of a block, by the note it was written with (blockNote adds how to retry; the note is found inside). */ +export function blockKind(note: string): BlockKind { + return BLOCK_NOTES.find(([, notes]) => notes.some((n) => note.includes(n)))?.[0] ?? "other"; +} + +/** The rows the report is built from. `moves` is every status move (a change merged now may have been accepted long ago); the rest are the window's. */ +export interface RetroInput { + days: number; + now: Date; + cfg: Config; + moves: StatusMove[]; + gates: GateRow[]; + phases: PhaseRow[]; + /** `question` events. */ + questions: EventRow[]; + /** `request` events: an applied answer ends a question's wait. */ + requests: EventRow[]; + judges: JudgeReport; +} + +/** One gate check over the window. People's own checks (see PERSON_CHECKS) are left out. */ +interface GateLine { gate: string; check: string; runs: number; passed: number; changes: number; firstTime: number } + +/** Waits at one review status a person decides, or for answers to questions (`at` is "question"); an open wait counts until now. */ +interface Wait { at: Status | "question"; waits: number; open: number; medianMs: number } + +/** + * What the suggestions read (see suggestions). Everything here is over the window. A rule that needs + * another fact adds it here and in buildRetro. + */ +export interface RetroFacts { + /** Blocks, by kind (every kind present, 0 when none). */ + blocks: Record<BlockKind, number>; + /** Changes that reached main. */ + merged: number; + /** What the loop spent, chat left out, and over how many changes. */ + spentUsd: number; + changesSpent: number; + /** What the judges' sessions (spec-check, plan-challenge, review, and their cheaper judges) cost. */ + judgesUsd: number; + /** The judges whose record has earned going first (judgeReport's `earned`). */ + judgesEarned: JudgeName[]; + gates: GateLine[]; + waits: Wait[]; +} + +/** One kind of block: how many, the changes (each with its count), and for other blocks each note's first sentence. */ +interface BlockGroup { kind: BlockKind; count: number; slugs: Array<[string, number]>; notes: Array<{ slug: string; note: string }> } + +/** One kind of step's spending. `changes`: how many changes ran it. */ +interface KindSpend { kind: string; usd: number; changes: number } + +/** The report: the facts, and what is only shown. */ +export interface Retro extends RetroFacts { + days: number; + since: string; + /** Nothing happened in the window (chat aside). */ + empty: boolean; + reached: string[]; + finished: string[]; + blocked: string[]; + closed: string[]; + /** From accepted to merged, for the changes that reached main; null when none did. */ + toMerged: { changes: number; medianMs: number } | null; + blockGroups: BlockGroup[]; + spending: { totalUsd: number; kinds: KindSpend[]; chatUsd: number }; + suggestions: string[]; +} + +/** The window's first moment: `days` before `now`, as an ISO time. */ +export function windowStart(now: Date, days: number): string { + return new Date(now.getTime() - days * DAY_MS).toISOString(); +} + +const HOUR_MS = 3_600_000; +const DAY_MS = 24 * HOUR_MS; + +/** Checks that record what a person did, not an automatic check. */ +const PERSON_CHECKS: ReadonlySet<string> = new Set(["done/human", "merge/pr-approved"]); + +/** The gates in the order a change meets them. */ +const GATE_ORDER = ["spec", "plan", "review", "merge", "done"]; + +/** The review statuses in the order a change meets them; their waits are listed so. */ +const WAIT_ORDER: ReadonlyArray<Status | "question"> = ["spec-review", "plan-review", "merge-review", "verifying", "question"]; + +/** How many kinds of step the spending lists. */ +const TOP_KINDS = 5; + +/** A request event's payload, as far as an answer's is read. */ +const RequestPayload = z.object({ kind: z.string(), result: z.string() }); + +function median(values: readonly number[]): number { + const v = [...values].sort((a, b) => a - b); + const mid = Math.floor(v.length / 2); + return v.length % 2 ? v[mid]! : (v[mid - 1]! + v[mid]!) / 2; +} + +/** Each slug once, in the order first seen. */ +const unique = (slugs: readonly string[]): string[] => [...new Set(slugs)]; + +/** Builds the report over the window from the trace's rows. */ +export function buildRetro(input: RetroInput): Retro { + const since = windowStart(input.now, input.days); + const nowIso = input.now.toISOString(); + // A move to the same status only rewrote the note. + const moves = input.moves.filter((m) => m.slug !== CHAT_SLUG && m.from !== m.to); + const inWindow = moves.filter((m) => m.ts >= since); + const to = (status: string) => unique(inWindow.filter((m) => m.to === status).map((m) => m.slug)); + const gates = input.gates.filter((g) => g.ts >= since && g.slug !== CHAT_SLUG); + const phases = input.phases.filter((p) => p.started >= since && p.status !== "running"); + const questions = input.questions.filter((q) => q.ts >= since && q.slug !== CHAT_SLUG); + const reached = to("merged"); + + const blockMoves = inWindow.filter((m) => m.to === "blocked"); + const blocks = Object.fromEntries(KINDS.map((k) => [k, 0])) as Record<BlockKind, number>; + for (const m of blockMoves) blocks[blockKind(m.note)]++; + + const spending = spend(phases); + const judging = phases.filter((p) => p.slug !== CHAT_SLUG && (JUDGES as readonly string[]).includes(phaseKind(p.name))); + const facts: RetroFacts = { + blocks, + merged: reached.length, + spentUsd: spending.totalUsd, + changesSpent: new Set(phases.filter((p) => p.slug !== CHAT_SLUG).map((p) => p.slug)).size, + judgesUsd: judging.reduce((a, p) => a + p.cost_usd, 0), + judgesEarned: input.judges.judges.filter((j) => j.earned).map((j) => j.judge), + gates: gateLines(gates), + waits: waits(moves, since, nowIso, questions, input.requests), + }; + return { + ...facts, + days: input.days, + since, + empty: !inWindow.length && !gates.length && !phases.some((p) => p.slug !== CHAT_SLUG) && !questions.length, + reached, + finished: to("done"), + blocked: unique(blockMoves.map((m) => m.slug)), + closed: to("closed"), + toMerged: toMerged(moves, reached, since), + blockGroups: blockGroups(blockMoves), + spending: { ...spending, kinds: spending.kinds.slice(0, TOP_KINDS) }, + suggestions: suggestions(facts, input.cfg), + }; +} + +/** + * From accepted to merged, for each change that reached main in the window: from its move to accepted, + * or, for a change written as accepted, its first move (out of accepted). + */ +function toMerged(moves: readonly StatusMove[], reached: readonly string[], since: string): Retro["toMerged"] { + const spans: number[] = []; + for (const slug of reached) { + const mine = moves.filter((m) => m.slug === slug); + const merged = mine.find((m) => m.to === "merged" && m.ts >= since); + const start = mine.find((m) => m.to === "accepted") ?? mine.find((m) => m.from !== "draft"); + if (merged && start && start.ts <= merged.ts) spans.push(Date.parse(merged.ts) - Date.parse(start.ts)); + } + return spans.length ? { changes: spans.length, medianMs: median(spans) } : null; +} + +/** The first sentence of a note: what happened, without the advice after it. */ +function firstSentence(note: string): string { + return /^.*?[.!?](?=\s|$)/.exec(note.trim())?.[0] ?? note.trim(); +} + +function blockGroups(blockMoves: readonly StatusMove[]): BlockGroup[] { + const groups = new Map<BlockKind, BlockGroup>(); + for (const m of blockMoves) { + const kind = blockKind(m.note); + const g = groups.get(kind) ?? { kind, count: 0, slugs: [], notes: [] }; + groups.set(kind, g); + g.count++; + const seen = g.slugs.find(([s]) => s === m.slug); + if (seen) seen[1]++; + else g.slugs.push([m.slug, 1]); + const note = firstSentence(m.note); + if (kind === "other" && !g.notes.some((n) => n.slug === m.slug && n.note === note)) g.notes.push({ slug: m.slug, note }); + } + return [...groups.values()].sort((a, b) => b.count - a.count || KINDS.indexOf(a.kind) - KINDS.indexOf(b.kind)); +} + +function gateLines(gates: readonly GateRow[]): GateLine[] { + const lines = new Map<string, GateLine & { seen: Set<string> }>(); + for (const g of gates) { + const key = `${g.gate}/${g.check}`; + if (PERSON_CHECKS.has(key)) continue; + const line = lines.get(key) ?? { gate: g.gate, check: g.check, runs: 0, passed: 0, changes: 0, firstTime: 0, seen: new Set<string>() }; + lines.set(key, line); + line.runs++; + if (g.result === "pass") line.passed++; + if (!line.seen.has(g.slug)) { + line.seen.add(g.slug); + line.changes++; + if (g.result === "pass") line.firstTime++; + } + } + const order = (gate: string) => { const i = GATE_ORDER.indexOf(gate); return i < 0 ? GATE_ORDER.length : i; }; + return [...lines.values()].sort((a, b) => order(a.gate) - order(b.gate)).map(({ seen: _, ...line }) => line); +} + +/** The window's ended phases' cost: the loop's in all and by kind of step (biggest first), and chat's. */ +function spend(phases: readonly PhaseRow[]): { totalUsd: number; kinds: KindSpend[]; chatUsd: number } { + const kinds = new Map<string, { usd: number; slugs: Set<string> }>(); + let totalUsd = 0; + let chatUsd = 0; + for (const p of phases) { + if (p.slug === CHAT_SLUG) { chatUsd += p.cost_usd; continue; } + totalUsd += p.cost_usd; + const kind = phaseKind(p.name); + const k = kinds.get(kind) ?? { usd: 0, slugs: new Set<string>() }; + kinds.set(kind, k); + k.usd += p.cost_usd; + k.slugs.add(p.slug); + } + const listed = [...kinds.entries()].filter(([, k]) => k.usd > 0).map(([kind, k]) => ({ kind, usd: k.usd, changes: k.slugs.size })); + return { totalUsd, kinds: listed.sort((a, b) => b.usd - a.usd), chatUsd }; +} + +/** + * Waits in the window. At a review status: from a move to it (in the window) to the change's next move + * to another status. merge-review waits for a person unless its note says it waits for GitHub's checks + * (nobody on the gate); the other review statuses are only entered when a person decides. For a + * question: from the question to an applied answer, or to the change's next status move. + */ +function waits(moves: readonly StatusMove[], since: string, now: string, questions: readonly EventRow[], requests: readonly EventRow[]): Wait[] { + const spans = new Map<Status | "question", { ms: number[]; open: number }>(); + const add = (at: Status | "question", start: string, end: string | undefined) => { + const w = spans.get(at) ?? { ms: [], open: 0 }; + spans.set(at, w); + w.ms.push(Date.parse(end ?? now) - Date.parse(start)); + if (end === undefined) w.open++; + }; + moves.forEach((m, i) => { + if (m.ts < since || !(m.to in REVIEW_GATE)) return; + if (m.to === "merge-review" && m.note === PR_CHECKS_NOTE) return; + add(m.to as Status, m.ts, moves.slice(i + 1).find((n) => n.slug === m.slug)?.ts); + }); + const answers = requests.filter((r) => { + let raw: unknown; + try { raw = JSON.parse(r.payload); } catch { return false; } + const p = RequestPayload.safeParse(raw); + return p.success && p.data.kind === "answer" && p.data.result === "applied"; + }); + for (const q of questions) { + const ends = [...answers.filter((a) => a.slug === q.slug), ...moves.filter((m) => m.slug === q.slug)].map((e) => e.ts).filter((ts) => ts > q.ts).sort(); + add("question", q.ts, ends[0]); + } + return WAIT_ORDER.flatMap((at) => { + const w = spans.get(at); + return w ? [{ at, waits: w.ms.length, open: w.open, medianMs: median(w.ms) }] : []; + }); +} + +/** Suggestion thresholds. */ +const JUDGING_SHARE = 0.3; +const JUDGING_CHANGES = 5; +const FIX_LOOP_BLOCKS = 2; +const TIMEOUT_BLOCKS = 2; +const TIMEOUT_GROWTH = 1.5; +const REVIEW_BLOCKS = 2; +const BUDGET_BLOCKS = 1; +const LONG_WAIT_MS = DAY_MS; +const QUIET_GATE_RUNS = 5; + +const money = (usd: number) => `$${usd.toFixed(2)}`; +const count = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`; + +/** + * What to change, from the facts: each one sentence with its evidence and the setting. Empty when no + * rule applies. Thresholds are the constants above. + */ +export function suggestions(f: RetroFacts, cfg: Config): string[] { + const out: string[] = []; + const setting = cfg.speed.cheap_judges; + if (setting === "shadow" && f.judgesEarned.length) { + const n = f.judgesEarned.length; + out.push(`The cheaper ${andList(f.judgesEarned)} judge${n === 1 ? " has" : "s have"} earned going first (see loopstra judges): set speed.cheap_judges to first to spend less on judging.`); + } + if (setting === "off" && f.spentUsd > 0 && f.judgesUsd / f.spentUsd >= JUDGING_SHARE && f.changesSpent >= JUDGING_CHANGES) { + out.push(`Judging cost ${money(f.judgesUsd)} of the ${money(f.spentUsd)} spent (${Math.round((f.judgesUsd / f.spentUsd) * 100)}%) over ${f.changesSpent} changes: set speed.cheap_judges to shadow to measure whether a cheaper model judges as well.`); + } + if (f.blocks.tests >= FIX_LOOP_BLOCKS) { + const loops = cfg.stages.build.max_fix_loops; + out.push(`${count(f.blocks.tests, "block")} came from tests that kept failing: raise stages.build.max_fix_loops from ${loops} to ${loops + 1}, or write smaller changes.`); + } + if (f.blocks.timeout >= TIMEOUT_BLOCKS) { + const minutes = cfg.claude.timeout_minutes; + out.push(`${count(f.blocks.timeout, "step")} timed out: raise claude.timeout_minutes from ${minutes} to ${Math.ceil((minutes * TIMEOUT_GROWTH) / 5) * 5}, or write smaller changes.`); + } + if (f.blocks.review >= REVIEW_BLOCKS) { + const rounds = cfg.stages.review.max_rounds; + out.push(`${count(f.blocks.review, "block")} came from a reviewer that kept finding problems: raise stages.review.max_rounds from ${rounds} to ${rounds + 1}.`); + } + if (f.blocks.budget >= BUDGET_BLOCKS) { + const limit = cfg.claude.max_budget_usd; + out.push(`${count(f.blocks.budget, "step")} hit the session spending limit${limit === undefined ? "" : ` of ${money(limit)}`}: raise claude.max_budget_usd with loopstra setup budgets.`); + } + const human = humanGates(cfg); + for (const w of f.waits) { + const gate = w.at === "question" ? undefined : REVIEW_GATE[w.at]; + if (!gate || human[gate] === "none" || w.medianMs <= LONG_WAIT_MS) continue; + const checks = f.gates.filter((g) => g.gate === gate); + const runs = checks.reduce((a, g) => a + g.runs, 0); + if (runs < QUIET_GATE_RUNS || checks.some((g) => g.passed < g.runs)) continue; + out.push(`Changes waited ${duration(w.medianMs)} at the median in ${w.at}, and the ${gate} gate's automatic checks passed all ${runs} times: the person on it may not be needed (gates.${gate}.human: none).`); + } + // Two more rules come with the settings they name, once those exist: + // - risky changes: gates.merge.human none, nothing set under gates.merge.risky, f.merged >= 1; + // - acceptance tests: stages.build.acceptance_tests off, f.blocks.unmet >= 1. + return out; +} + +/** A length of time in words: minutes under an hour, hours under two days, then days. */ +function duration(ms: number): string { + const round = (n: number) => Math.round(n * 10) / 10; + if (ms < HOUR_MS) return count(Math.max(1, Math.round(ms / 60_000)), "min", "min"); + if (ms < 2 * DAY_MS) return count(round(ms / HOUR_MS), "hour"); + return count(round(ms / DAY_MS), "day"); +} + +/** Up to five changes, then how many more. */ +function names(slugs: readonly string[]): string { + const shown = slugs.slice(0, 5); + return andList(slugs.length > shown.length ? [...shown, `${slugs.length - shown.length} more`] : shown); +} + +/** "add-numbers", "add-numbers twice", "add-numbers 3 times". */ +function times([slug, n]: [string, number]): string { + return n === 1 ? slug : n === 2 ? `${slug} twice` : `${slug} ${n} times`; +} + +/** A gate check in words: the check alone when it already names its gate ("spec-check"), else both ("review findings"). */ +function checkName(g: GateLine): string { + return g.check.startsWith(g.gate) ? g.check : `${g.gate} ${g.check}`; +} + +/** "Oct 20": a day in local time. */ +const day = (iso: string) => new Date(iso).toLocaleDateString("en-US", { month: "short", day: "numeric" }); + +/** The report as plain text. */ +export function retroText(r: Retro): string { + const span = r.days === 1 ? "last day" : `last ${r.days} days`; + const tail = ["", "Suggestions", ...(r.suggestions.length ? r.suggestions : ["No suggestions: nothing in the trace points at a setting to change."]).map((s) => `- ${s}`)]; + if (r.empty) return [`Nothing happened in the ${span}.`, ...tail].join("\n") + "\n"; + const listed = (label: string, slugs: readonly string[]) => `- ${label}: ${slugs.length}${slugs.length ? ` (${names(slugs)})` : ""}.`; + const lines = [ + `The ${span} (since ${day(r.since)}).`, + "", "What happened", + listed("Reached main", r.reached), listed("Finished", r.finished), listed("Blocked", r.blocked), listed("Closed", r.closed), + ...(r.toMerged ? [`- From accepted to merged: ${duration(r.toMerged.medianMs)} at the median, over ${count(r.toMerged.changes, "change")}.`] : []), + "", "Where changes blocked", + ...(r.blockGroups.length ? r.blockGroups.flatMap((g) => [ + `- ${KIND_WORDS[g.kind]}: ${g.count} (${names(g.slugs.map(times))}).`, + ...g.notes.slice(0, 5).map((n) => ` - ${n.slug}: ${n.note}`), + ]) : ["- Nothing blocked."]), + "", "Gates", + ...(r.gates.length ? r.gates.map((g) => `- ${checkName(g)}: ran ${count(g.runs, "time")}; ${g.firstTime} of ${count(g.changes, "change")} passed the first time.`) : ["- No automatic checks ran."]), + "", "Spending", + ...spendingLines(r), + "", "Waiting for people", + ...(r.waits.length ? r.waits.map((w) => `- ${w.at === "question" ? "Questions" : w.at}: ${count(w.waits, "wait")}, ${duration(w.medianMs)} at the median${w.open ? ` (${w.open} still waiting)` : ""}.`) : ["- Nobody was waited for."]), + ...tail, + ]; + return lines.join("\n") + "\n"; +} + +function spendingLines(r: Retro): string[] { + const { totalUsd, kinds, chatUsd } = r.spending; + const chat = chatUsd > 0 ? [`- Chat: ${money(chatUsd)}, not counted above.`] : []; + if (totalUsd <= 0) return [...(chat.length ? [] : ["- Nothing was spent."]), ...chat]; + return [ + `- ${money(totalUsd)} in all, over ${count(r.changesSpent, "change")} (${money(totalUsd / r.changesSpent)} a change).`, + ...kinds.map((k) => `- ${k.kind}: ${money(k.usd)} (${Math.round((k.usd / totalUsd) * 100)}%), ${money(k.usd / k.changes)} a change.`), + ...chat, + ]; +} diff --git a/src/stages/build.ts b/src/stages/build.ts index 5beadbed..f508464c 100644 --- a/src/stages/build.ts +++ b/src/stages/build.ts @@ -108,7 +108,7 @@ export async function runBuildStep(ctx: StepContext): Promise<StepResult> { const again = await verifyChange(ctx, a.spec, plan, "verify-2"); if (!again.ok) return block(ctx, again.note, again); if (!again.passed) { - return block(ctx, `The finished change still did not work as the spec describes after one round of fixes. An engineer needs to look at it.`, { detail: { observations: again.observations } }); + return block(ctx, VERIFY_STILL_FAILED_NOTE, { detail: { observations: again.observations } }); } } @@ -137,6 +137,13 @@ async function reviewsAhead(ctx: StepContext, wt: Git): Promise<boolean> { } } +/** The owner's notes when the checks still fail after every fix allowed (see testLoop): the tests, or a command that kept running past its limit. */ +export const TESTS_KEPT_FAILING_NOTE = "The tests kept failing after several attempts to fix them. An engineer needs to look at the change."; +export const CHECKS_KEPT_TIMING_OUT_NOTE = "A project command kept running past its time limit, even after attempts to fix the change. An engineer needs to look at it."; + +/** The owner's note when the verifier still finds the change does not work after its round of fixes. */ +export const VERIFY_STILL_FAILED_NOTE = "The finished change still did not work as the spec describes after one round of fixes. An engineer needs to look at it."; + /** What fix is told after a verify failure: the checks passed, so it must not read as a failed check. */ const VERIFY_FAILED = "The checks pass; the verifier found the change does not do what the spec says:"; @@ -185,9 +192,7 @@ export async function testLoop(ctx: StepContext, prefix: string): Promise<{ ok: return { ok: true }; } if (i > max) { - const note = failure.timedOut - ? "A project command kept running past its time limit, even after attempts to fix the change. An engineer needs to look at it." - : "The tests kept failing after several attempts to fix them. An engineer needs to look at the change."; + const note = failure.timedOut ? CHECKS_KEPT_TIMING_OUT_NOTE : TESTS_KEPT_FAILING_NOTE; return { ok: false, note, detail: `${failure.command} still failing after ${max} fix attempts: ${failure.lastLine}` }; } const fix = await buildSession(ctx, { diff --git a/src/stages/merge.ts b/src/stages/merge.ts index 0bfe6241..312d490b 100644 --- a/src/stages/merge.ts +++ b/src/stages/merge.ts @@ -15,12 +15,12 @@ import { REVIEW_PASSED, runReviewRounds } from "./review"; import { alreadyTested, KEEP_UNFINISHED, MERGE_WAIT_NOTE, MERGING, openBranchWorktree, PR_APPROVE_NOTE, PR_CHECKS_NOTE, PR_LEASE, PR_MERGED_ELSEWHERE, PR_NUMBER, PR_PUSHING, PR_WAIT, readRound, REMOTE_AHEAD, REVIEW_ROUND, saveWork, writeRound } from "./shared"; export const DIRTY_ROOT_NOTE = "The main checkout has unsaved changes or is on another branch; an engineer needs to tidy it up before this can merge. Then set status to merge-approved."; -const UPDATE_FAILED_NOTE = "The change overlaps with other recent changes and could not be brought up to date automatically. Nothing was merged. An engineer needs to look at it."; -const REVIEWS_USED_UP = "The change needed more fixes after it was reviewed, and it has already been reviewed as many times as allowed. An engineer needs to look at the change."; +export const UPDATE_FAILED_NOTE = "The change overlaps with other recent changes and could not be brought up to date automatically. Nothing was merged. An engineer needs to look at it."; +export const REVIEWS_USED_UP = "The change needed more fixes after it was reviewed, and it has already been reviewed as many times as allowed. An engineer needs to look at the change."; /** Notes for the pull request path (a remote exists). */ -const PR_CLOSED_NOTE = "The pull request was closed without merging. Set status to closed, or to plan-approved to rebuild."; -const PR_CHECKS_FAILED_NOTE = "The automatic checks on GitHub failed. An engineer should look at the pull request."; +export const PR_CLOSED_NOTE = "The pull request was closed without merging. Set status to closed, or to plan-approved to rebuild."; +export const PR_CHECKS_FAILED_NOTE = "The automatic checks on GitHub failed. An engineer should look at the pull request."; const PR_BRANCH_MOVED_NOTE = "Someone added commits to the pull request's branch on GitHub, and the branch here changed too, so Loopstra overwrote neither. An engineer needs to bring the two together."; const PERSON_COMMITS_NOT_TAKEN_NOTE = "Someone added commits to the pull request's branch on GitHub, and they could not be taken into the workspace here, so nothing was merged. An engineer needs to look at the change's workspace (for example unsaved edits to the same files)."; const PR_MERGED_WITHOUT_FIXES_NOTE = "The pull request was merged on GitHub before this change's latest fixes were made, so main does not have them; they are still on the change's branch. An engineer needs to look at the branch. Then set status to closed, or to plan-approved to rebuild."; diff --git a/src/stages/review.ts b/src/stages/review.ts index 7fc7b5af..762f9f72 100644 --- a/src/stages/review.ts +++ b/src/stages/review.ts @@ -10,6 +10,9 @@ import { artifacts, bullets, buildSession, KEEP_UNFINISHED, openBranchWorktree, /** The review rounds ended with a review that had no important findings (and the after commands ran). */ export const REVIEW_PASSED = "review-passed"; +/** The owner's note when the review rounds run out with important findings left. */ +export const REVIEW_STILL_FOUND_NOTE = "The reviewer still found important problems after the change was revised. The details are in review.md. An engineer needs to look at the change."; + /** * Stage 5 review rounds, run by the merge step for reviewing (see runMergeStep): review, and on * important findings revise and test again, until a review passes or the rounds run out (block). @@ -63,7 +66,7 @@ export async function runReviewRounds(ctx: StepContext): Promise<StepResult | ty if (round > stage.max_rounds) { clearMarker(ctx, REVIEW_ROUND); - return block(ctx, "The reviewer still found important problems after the change was revised. The details are in review.md. An engineer needs to look at the change.", { detail: { rounds: round, important } }); + return block(ctx, REVIEW_STILL_FOUND_NOTE, { detail: { rounds: round, important } }); } const revise = await buildSession(ctx, { name: "revise", traceName: `revise-${round}`, permissionMode: "acceptEdits", tools: "build", cwd: ctx.worktreeDir, diff --git a/src/stages/verify.ts b/src/stages/verify.ts index 2aee5d89..1bfb5a42 100644 --- a/src/stages/verify.ts +++ b/src/stages/verify.ts @@ -10,7 +10,7 @@ import { errorText } from "../shell"; const JUDGE_FAILED = "The final check could not be completed; an engineer should look."; export const NEEDS_PERSON_DONE_NOTE = "Done. A few results need a person to confirm; see outcome.md."; -const UNMET_NOTE = "The change merged, but the final check found Done when criteria that are not met. The details are in outcome.md. Decide whether to open a follow-up change, then set this one to done or closed."; +export const UNMET_NOTE = "The change merged, but the final check found Done when criteria that are not met. The details are in outcome.md. Decide whether to open a follow-up change, then set this one to done or closed."; const NO_CHECK_OUTCOME = "# Outcome\n\n## Outcome\nThe change is merged.\n\n## Evidence\nNo automatic check of the Done when criteria is configured. A person may want to confirm them.\n"; diff --git a/tests/unit/cli.test.ts b/tests/unit/cli.test.ts index 0c1c81b9..d2d41315 100644 --- a/tests/unit/cli.test.ts +++ b/tests/unit/cli.test.ts @@ -11,10 +11,10 @@ const CLI = fileURLToPath(new URL("../../src/cli.ts", import.meta.url)); const NOT_SET_UP = "This folder is not set up for Loopstra. Run loopstra init first."; describe("the command line outside a set-up repo", () => { - test("status, tail, ui, judges, upgrade, and customize say the folder is not set up, and create nothing", async () => { + test("status, tail, ui, judges, retro, upgrade, and customize say the folder is not set up, and create nothing", async () => { const t = tempDir(); try { - for (const cmd of ["status", "tail", "ui", "judges", "upgrade", "customize"]) { + for (const cmd of ["status", "tail", "ui", "judges", "retro", "upgrade", "customize"]) { const r = await run([process.execPath, CLI, cmd], t.path); expect({ cmd, code: r.code, said: (r.out + r.err).trim() }).toEqual({ cmd, code: 1, said: NOT_SET_UP }); } @@ -24,10 +24,12 @@ describe("the command line outside a set-up repo", () => { } }); - test("help lists judges", async () => { + test("help lists judges and retro", async () => { const t = tempDir(); try { - expect((await run([process.execPath, CLI, "help"], t.path)).out).toContain("\n judges how often the cheaper judges agreed with the strong ones\n"); + const out = (await run([process.execPath, CLI, "help"], t.path)).out; + expect(out).toContain("\n judges how often the cheaper judges agreed with the strong ones\n"); + expect(out).toContain("\n retro how the loop did lately, and what to change (--days <n>, default 7)\n"); } finally { t.cleanup(); } @@ -119,6 +121,10 @@ describe("the command line outside a set-up repo", () => { [["tail", "one", "two"], "Usage: loopstra tail [slug]"], [["tail", "--follow"], "Usage: loopstra tail [slug]"], [["judges", "--all"], "Usage: loopstra judges"], + [["retro", "--days", "0"], "Usage: loopstra retro [--days <number>]"], + [["retro", "--days", "a week"], "Usage: loopstra retro [--days <number>]"], + [["retro", "--days"], "Usage: loopstra retro [--days <number>]"], + [["retro", "7"], "Usage: loopstra retro [--days <number>]"], [["upgrade", "--force"], "Usage: loopstra upgrade [--check] [--repo]"], [["customize", "--list"], "Usage: loopstra customize [prompt]"], [["apply-lessons", "one", "--dry"], "Usage: loopstra apply-lessons <slug>"], @@ -149,6 +155,21 @@ describe("the command line outside a set-up repo", () => { }); }); +describe("loopstra retro in a set-up repo", () => { + test("prints the report for the days asked, and exits 0 on a brand-new trace", async () => { + const t = tempDir(); + try { + mkdirSync(join(t.path, "loopstra"), { recursive: true }); + writeFileSync(join(t.path, "loopstra", "config.yaml"), "version: 1\ncommands:\n test: echo ok\n"); + const r = await run([process.execPath, CLI, "retro", "--days", "3"], t.path); + expect({ code: r.code, err: r.err }).toEqual({ code: 0, err: "" }); + expect(r.out).toStartWith("Nothing happened in the last 3 days.\n"); + } finally { + t.cleanup(); + } + }); +}); + describe("the update notice", () => { test("status in a repository set up before the stamp says to upgrade, even from a development checkout; with a stamp it says nothing", async () => { const repo = await tempGitRepo(); @@ -172,10 +193,10 @@ describe("parseArgs", () => { const ui = { usage: "Usage: loopstra ui [--port <number>] [--no-chat]", flags: ["--no-chat"], port: true } as const; test("the flags a command takes, in either order, with or without a port", () => { - expect(parseArgs([], ui)).toEqual({ flags: new Set(), port: null, names: [] }); - expect(parseArgs(["--no-chat"], ui)).toEqual({ flags: new Set(["--no-chat"]), port: null, names: [] }); - expect(parseArgs(["--port", "5000", "--no-chat"], ui)).toEqual({ flags: new Set(["--no-chat"]), port: 5000, names: [] }); - expect(parseArgs(["--no-chat", "--port=5000"], ui)).toEqual({ flags: new Set(["--no-chat"]), port: 5000, names: [] }); + expect(parseArgs([], ui)).toEqual({ flags: new Set(), port: null, days: null, names: [] }); + expect(parseArgs(["--no-chat"], ui)).toEqual({ flags: new Set(["--no-chat"]), port: null, days: null, names: [] }); + expect(parseArgs(["--port", "5000", "--no-chat"], ui)).toEqual({ flags: new Set(["--no-chat"]), port: 5000, days: null, names: [] }); + expect(parseArgs(["--no-chat", "--port=5000"], ui)).toEqual({ flags: new Set(["--no-chat"]), port: 5000, days: null, names: [] }); expect(parseArgs(["--port=0"], ui)!.port).toBe(0); expect(parseArgs(["--port", "65535"], ui)!.port).toBe(65535); }); @@ -187,8 +208,20 @@ describe("parseArgs", () => { expect(parseArgs(["--port", "5000"], { usage: "u" })).toBeNull(); }); + test("days: a whole number from 1 to 3650, given once, only to a command that takes it", () => { + const retro = { usage: "Usage: loopstra retro [--days <number>]", days: true } as const; + expect(parseArgs([], retro)).toEqual({ flags: new Set(), port: null, days: null, names: [] }); + expect(parseArgs(["--days", "30"], retro)!.days).toBe(30); + expect(parseArgs(["--days=1"], retro)!.days).toBe(1); + expect(parseArgs(["--days", "3650"], retro)!.days).toBe(3650); + for (const bad of [["--days"], ["--days", "0"], ["--days", "-1"], ["--days", "1.5"], ["--days", "x"], ["--days=7d"], ["--days", "3651"], ["--days", "3", "--days", "4"], ["3"]]) { + expect({ bad, parsed: parseArgs(bad, retro) }).toEqual({ bad, parsed: null }); + } + expect(parseArgs(["--days", "3"], { usage: "u" })).toBeNull(); + }); + test("names: as many as the command takes", () => { - expect(parseArgs(["one"], { usage: "u", names: 1 })).toEqual({ flags: new Set(), port: null, names: ["one"] }); + expect(parseArgs(["one"], { usage: "u", names: 1 })).toEqual({ flags: new Set(), port: null, days: null, names: ["one"] }); expect(parseArgs(["one", "two"], { usage: "u", names: 1 })).toBeNull(); expect(parseArgs(["--check", "--repo"], { usage: "u", flags: ["--check", "--repo"] })!.flags).toEqual(new Set(["--check", "--repo"])); }); diff --git a/tests/unit/judges.test.ts b/tests/unit/judges.test.ts index 197d08f2..612c0c69 100644 --- a/tests/unit/judges.test.ts +++ b/tests/unit/judges.test.ts @@ -150,6 +150,14 @@ describe("judgeReport", () => { expect(report(rows(...caught(3)), "off")).toBeNull(); }); + test("earned says whether the record has earned going first, whatever the setting; never for a judge that cannot be compared", () => { + const earned = (seeded: JudgeRow[], setting = "shadow", yaml = "") => judgeReport(cfg(yaml, setting), seeded, TEMPLATES).judges.find((j) => j.judge === "review")!.earned; + expect(earned(rows(...caught(29)))).toBe(false); + expect(earned(rows(...caught(30)))).toBe(true); + expect(earned(rows(...caught(30)), "off")).toBe(true); + expect(earned(rows(...caught(30)), "shadow", "stages:\n review:\n model: default\n")).toBe(false); + }); + test("a judge that cannot be compared says why in words", () => { const why = (yaml: string, judge = "review") => judgeReport(cfg(yaml), [], TEMPLATES).judges.find((j) => j.judge === judge)!.why; expect(why("stages:\n review:\n model: default\n")).toBe("review already runs on sonnet; there is nothing cheaper to compare"); diff --git a/tests/unit/retro.test.ts b/tests/unit/retro.test.ts new file mode 100644 index 00000000..7b06306c --- /dev/null +++ b/tests/unit/retro.test.ts @@ -0,0 +1,408 @@ +import { describe, expect, test } from "bun:test"; +import { mkdirSync } from "node:fs"; +import { join } from "node:path"; +import { Database } from "bun:sqlite"; +import { renderRetro } from "../../src/commands/retro"; +import { configPath, parseConfig, type Config } from "../../src/config"; +import { CRASH_NOTE, TIMEOUT_NOTE } from "../../src/context"; +import { judgeReport, type JudgeReport } from "../../src/judges"; +import { ownerNote } from "../../src/phases"; +import { TOO_MANY_NOTE } from "../../src/questions"; +import { blockKind, buildRetro, retroText, suggestions, type BlockKind, type RetroInput } from "../../src/retro"; +import { CHECKS_KEPT_TIMING_OUT_NOTE, TESTS_KEPT_FAILING_NOTE, VERIFY_STILL_FAILED_NOTE } from "../../src/stages/build"; +import { PR_CHECKS_FAILED_NOTE, PR_CLOSED_NOTE, REVIEWS_USED_UP, UPDATE_FAILED_NOTE } from "../../src/stages/merge"; +import { REVIEW_STILL_FOUND_NOTE } from "../../src/stages/review"; +import { MERGE_WAIT_NOTE, PR_CHECKS_NOTE } from "../../src/stages/shared"; +import { UNMET_NOTE } from "../../src/stages/verify"; +import { Trace, type EventRow, type GateRow, type PhaseRow, type StatusMove } from "../../src/trace"; +import { tempDir } from "../helpers"; + +const HOUR = 3_600_000; +const DAY = 24 * HOUR; +/** Local noon, so the window's first day reads the same in every time zone. */ +const NOW = new Date(2026, 9, 27, 12); +const ago = (days: number, hours = 0) => new Date(NOW.getTime() - days * DAY + hours * HOUR).toISOString(); +const BASE = "version: 1\ncommands:\n test: echo ok\n"; +const cfgOf = (yaml = "") => parseConfig(BASE + yaml); + +function input(over: Partial<RetroInput> = {}): RetroInput { + const cfg = over.cfg ?? cfgOf(); + return { days: 7, now: NOW, cfg, moves: [], gates: [], phases: [], questions: [], requests: [], judges: judgeReport(cfg, [], {}), ...over }; +} + +const mv = (slug: string, ts: string, from: string, to: string, note = ""): StatusMove => ({ slug, ts, from, to, note }); +const gateRow = (slug: string, gate: string, check: string, result: string, ts = ago(1)): GateRow => ({ slug, gate, check, result, ts }); +let seq = 0; +const phase = (slug: string, name: string, cost: number, status = "success", started = ago(1)): PhaseRow => + ({ slug, seq: ++seq, name, kind: "agent", status, started, ended: status === "running" ? null : started, cost_usd: cost, session_id: null, error: null }); +let id = 0; +const ev = (slug: string, type: EventRow["type"], ts: string, payload: unknown): EventRow => ({ id: ++id, slug, phase_seq: null, type, ts, payload: JSON.stringify(payload) }); +const blocks = (kind: BlockKind, n: number, slug = "slow-one"): StatusMove[] => { + const note: Record<string, string> = { tests: TESTS_KEPT_FAILING_NOTE, timeout: ownerNote("timeout"), review: REVIEW_STILL_FOUND_NOTE, budget: ownerNote("budget"), unmet: UNMET_NOTE }; + return Array.from({ length: n }, (_, i) => mv(slug, ago(2, i), "building", "blocked", note[kind])); +}; + +describe("blockKind", () => { + test("each note the runtime blocks with is told by the constant it is written from", () => { + const cases: Array<[string, BlockKind]> = [ + [ownerNote("timeout"), "timeout"], + [`${TIMEOUT_NOTE} To try again, set status to plan-approved.`, "timeout"], + [`The finished change could not be checked. ${ownerNote("timeout")}`, "timeout"], + [ownerNote("crash"), "crash"], + [`${CRASH_NOTE} To try again, set status to plan-approved.`, "crash"], + [ownerNote("budget"), "budget"], + [TOO_MANY_NOTE, "questions"], + [`${TESTS_KEPT_FAILING_NOTE} When that is sorted out, set status to plan-approved to try again.`, "tests"], + [CHECKS_KEPT_TIMING_OUT_NOTE, "tests"], + [REVIEW_STILL_FOUND_NOTE, "review"], + [REVIEWS_USED_UP, "review"], + [VERIFY_STILL_FAILED_NOTE, "verify"], + [UPDATE_FAILED_NOTE, "conflicts"], + [PR_CHECKS_FAILED_NOTE, "pr-checks"], + [PR_CLOSED_NOTE, "pr-closed"], + [UNMET_NOTE, "unmet"], + ["The work for this change is missing. To build it again, set status to plan-approved.", "other"], + ["", "other"], + ]; + for (const [note, kind] of cases) expect({ note, kind: blockKind(note) }).toEqual({ note, kind }); + }); +}); + +describe("buildRetro", () => { + test("what happened: changes that reached main, finished, blocked (once per change) and closed in the window, and the median from accepted to merged", () => { + const r = buildRetro(input({ + moves: [ + mv("old-one", ago(20), "draft", "accepted"), // accepted before the window + mv("ancient", ago(10), "merge-approved", "merged"), // merged before the window + mv("add-numbers", ago(6), "draft", "accepted"), + mv("add-numbers", ago(6, 2), "accepted", "designing"), + mv("fix-typo", ago(4), "accepted", "designing"), // written as accepted: the first move starts it + mv("add-numbers", ago(5), "merge-approved", "merged"), + mv("fix-typo", ago(4, 3), "merge-approved", "merged"), + mv("add-numbers", ago(5, 1), "merged", "verifying"), + mv("add-numbers", ago(5, 2), "verifying", "done"), + ...blocks("tests", 2), + mv("slow-one", ago(1), "blocked", "blocked", "a note rewritten"), + mv("old-one", ago(2), "merge-approved", "merged"), + mv("gone", ago(1), "spec-review", "closed"), + ], + })); + expect(r.reached).toEqual(["add-numbers", "fix-typo", "old-one"]); + expect(r.finished).toEqual(["add-numbers"]); + expect(r.blocked).toEqual(["slow-one"]); + expect(r.closed).toEqual(["gone"]); + expect(r.merged).toBe(3); + // 24 hours, 3 hours, 18 days: the median is 24 hours. + expect(r.toMerged).toEqual({ changes: 3, medianMs: 24 * HOUR }); + }); + + test("where changes blocked: grouped by kind, most common first (ties in the kinds' order), with the changes and other notes", () => { + const r = buildRetro(input({ + moves: [ + ...blocks("tests", 2), + mv("odd-one", ago(1), "merge-approved", "blocked", "The work for this change is missing. To build it again, set status to plan-approved."), + ...blocks("timeout", 1, "x"), + ...blocks("tests", 1, "other"), + ], + })); + expect(r.blockGroups).toEqual([ + { kind: "tests", count: 3, slugs: [["slow-one", 2], ["other", 1]], notes: [] }, + { kind: "timeout", count: 1, slugs: [["x", 1]], notes: [] }, + { kind: "other", count: 1, slugs: [["odd-one", 1]], notes: [{ slug: "odd-one", note: "The work for this change is missing." }] }, + ]); + expect(r.blocks.tests).toBe(3); + expect(r.blocks.crash).toBe(0); + }); + + test("gates: runs and first-time passes per check, people's own checks left out, in the stages' order", () => { + const r = buildRetro(input({ + gates: [ + gateRow("b", "review", "findings", "pass"), + gateRow("a", "spec", "spec-check", "fail"), + gateRow("a", "spec", "spec-check", "pass"), + gateRow("b", "spec", "spec-check", "pass"), + gateRow("a", "done", "human", "waiting"), + gateRow("a", "merge", "pr-approved", "pass"), + gateRow("a", "done", "done-check", "pass"), + ], + })); + expect(r.gates).toEqual([ + { gate: "spec", check: "spec-check", runs: 3, passed: 2, changes: 2, firstTime: 1 }, + { gate: "review", check: "findings", runs: 1, passed: 1, changes: 1, firstTime: 1 }, + { gate: "done", check: "done-check", runs: 1, passed: 1, changes: 1, firstTime: 1 }, + ]); + }); + + test("spending: phase names folded to their kind, the five biggest, chat on its own, running phases' holds left out", () => { + const r = buildRetro(input({ + phases: [ + phase("a", "build", 2), phase("a", "fix-1", 0.5), phase("a", "fix-2", 0.5), phase("a", "early-review", 0.25), + phase("a", "review-1", 0.75), phase("a", "spec-check-cheap", 0.1), phase("a", "test-1", 0), phase("b", "build", 1), + phase("b", "design", 0.05), phase("b", "plan", 0.04), phase("b", "intake", 0.01), + phase("c", "build", 5, "running"), phase("_chat", "chat", 0.4), + ], + })); + expect(r.spending.totalUsd).toBeCloseTo(5.2); + expect(r.spentUsd).toBeCloseTo(5.2); + expect(r.changesSpent).toBe(2); // c only holds part of a budget so far + expect(r.spending.chatUsd).toBeCloseTo(0.4); + expect(r.spending.kinds.map((k) => [k.kind, Number(k.usd.toFixed(2)), k.changes])).toEqual([ + ["build", 3, 2], ["fix", 1, 1], ["review", 1, 1], ["spec-check", 0.1, 1], ["design", 0.05, 1], + ]); + expect(r.judgesUsd).toBeCloseTo(1.1); + }); + + test("waiting for people: review statuses a person decides and questions, with the median wait; a wait still open counts until now", () => { + const r = buildRetro(input({ + moves: [ + mv("a", ago(5), "reviewing", "merge-review", MERGE_WAIT_NOTE), + mv("a", ago(5, 1), "merge-review", "merge-review", "the note rewritten"), + mv("a", ago(5, 30), "merge-review", "merge-approved"), + mv("b", ago(4), "reviewing", "merge-review", PR_CHECKS_NOTE), // waits for GitHub's checks, not a person + mv("b", ago(4, 1), "merge-review", "merged"), + mv("c", ago(2), "designing", "spec-review"), // still waiting: 2 days so far + mv("d", ago(9), "designing", "spec-review"), // began before the window + mv("d", ago(1), "spec-review", "spec-approved"), + mv("e", ago(4), "building", "blocked"), + ], + questions: [ev("e", "question", ago(3), { question: "Which?" }), ev("f", "question", ago(3), { question: "Where?" }), ev("_chat", "question", ago(3), {})], + requests: [ + ev("f", "request", ago(3, 1), { kind: "answer", result: "applied" }), + ev("e", "request", ago(3, 2), { kind: "answer", result: "refused" }), + ], + })); + expect(r.waits).toEqual([ + { at: "spec-review", waits: 1, open: 1, medianMs: 2 * DAY }, + { at: "merge-review", waits: 1, open: 0, medianMs: 30 * HOUR }, + // f answered after an hour; e's question waits until now (a refused answer, or a block before it, does not end it). + { at: "question", waits: 2, open: 1, medianMs: (1 * HOUR + 3 * DAY) / 2 }, + ]); + }); + + test("a status move ends a question's wait", () => { + const r = buildRetro(input({ + moves: [mv("e", ago(3, 3), "planning", "plan-review")], + questions: [ev("e", "question", ago(3), { question: "Which?" })], + })); + expect(r.waits.find((w) => w.at === "question")).toEqual({ at: "question", waits: 1, open: 0, medianMs: 3 * HOUR }); + }); + + test("an empty window, or one with only chat in it, is empty", () => { + expect(buildRetro(input()).empty).toBe(true); + expect(buildRetro(input({ phases: [phase("_chat", "chat", 1)], questions: [ev("_chat", "question", ago(1), {})] })).empty).toBe(true); + expect(buildRetro(input({ phases: [phase("a", "build", 1, "success", ago(9))] })).empty).toBe(true); + expect(buildRetro(input({ gates: [gateRow("a", "spec", "spec-check", "pass")] })).empty).toBe(false); + }); +}); + +describe("suggestions", () => { + const facts = (over: Partial<RetroInput> = {}) => buildRetro(input(over)); + const say = (over: Partial<RetroInput> = {}) => suggestions(facts(over), over.cfg ?? cfgOf()); + + test("none apply: none", () => { + expect(say()).toEqual([]); + }); + + test("cheaper judges: shadow with a judge that earned first suggests first; not when it has not, or the setting is first", () => { + const report = (cfg: Config, earned: string[]): JudgeReport => { + const r = judgeReport(cfg, [], {}); + return { ...r, judges: r.judges.map((j) => ({ ...j, earned: earned.includes(j.judge) })) }; + }; + const shadow = cfgOf("speed:\n cheap_judges: shadow\n"); + expect(say({ cfg: shadow, judges: report(shadow, ["spec-check", "review"]) })).toEqual([ + "The cheaper spec-check and review judges have earned going first (see loopstra judges): set speed.cheap_judges to first to spend less on judging.", + ]); + expect(say({ cfg: shadow, judges: report(shadow, ["review"]) })[0]).toStartWith("The cheaper review judge has earned going first"); + expect(say({ cfg: shadow, judges: report(shadow, []) })).toEqual([]); + const first = cfgOf("speed:\n cheap_judges: first\n"); + expect(say({ cfg: first, judges: report(first, ["review"]) })).toEqual([]); + }); + + test("cheaper judges: off, with judging at least 30% of the spending over at least 5 changes, suggests shadow", () => { + const spend = (judging: number, changes = 5) => Array.from({ length: changes }, (_, i) => [phase(`c${i}`, "build", (10 - judging) / changes), phase(`c${i}`, "review-1", judging / changes)]).flat(); + expect(say({ phases: spend(3) })).toEqual([ + "Judging cost $3.00 of the $10.00 spent (30%) over 5 changes: set speed.cheap_judges to shadow to measure whether a cheaper model judges as well.", + ]); + expect(say({ phases: spend(2.9) })).toEqual([]); + expect(say({ phases: spend(3, 4) })).toEqual([]); + expect(say({ phases: spend(5), cfg: cfgOf("speed:\n cheap_judges: shadow\n") })).toEqual([]); + }); + + test("fix loops: at least 2 blocks from failing tests suggest one more fix loop", () => { + expect(say({ moves: blocks("tests", 2) })).toEqual([ + "2 blocks came from tests that kept failing: raise stages.build.max_fix_loops from 3 to 4, or write smaller changes.", + ]); + expect(say({ moves: blocks("tests", 1) })).toEqual([]); + }); + + test("timeouts: at least 2 suggest half as long again, rounded up to 5 minutes", () => { + expect(say({ moves: blocks("timeout", 2) })).toEqual(["2 steps timed out: raise claude.timeout_minutes from 30 to 45, or write smaller changes."]); + expect(say({ moves: blocks("timeout", 2), cfg: cfgOf("claude:\n timeout_minutes: 45\n") })[0]).toContain("from 45 to 70"); + expect(say({ moves: blocks("timeout", 1) })).toEqual([]); + }); + + test("review rounds: at least 2 blocks from the reviewer suggest one more round", () => { + expect(say({ moves: blocks("review", 2) })).toEqual([ + "2 blocks came from a reviewer that kept finding problems: raise stages.review.max_rounds from 2 to 3.", + ]); + expect(say({ moves: blocks("review", 1) })).toEqual([]); + }); + + test("spending limit: one block on the session's limit suggests raising it", () => { + expect(say({ moves: blocks("budget", 1), cfg: cfgOf("claude:\n max_budget_usd: 2.5\n") })).toEqual([ + "1 step hit the session spending limit of $2.50: raise claude.max_budget_usd with loopstra setup budgets.", + ]); + expect(say({ moves: blocks("budget", 2) })).toEqual([ + "2 steps hit the session spending limit: raise claude.max_budget_usd with loopstra setup budgets.", + ]); + expect(say()).toEqual([]); + }); + + test("waits: a person's gate with a median wait over a day, whose automatic checks all passed at least 5 times, may not need the person", () => { + const merge = cfgOf("gates:\n merge:\n human: status\n"); + const waits = (hours: number) => [mv("a", ago(5), "reviewing", "merge-review", MERGE_WAIT_NOTE), mv("a", ago(5, hours), "merge-review", "merge-approved")]; + const checks = (n: number, last = "pass") => Array.from({ length: n }, (_, i) => gateRow(`c${i}`, "merge", "tests", i === n - 1 ? last : "pass")); + expect(say({ cfg: merge, moves: waits(25), gates: checks(5) })).toEqual([ + "Changes waited 25 hours at the median in merge-review, and the merge gate's automatic checks passed all 5 times: the person on it may not be needed (gates.merge.human: none).", + ]); + expect(say({ cfg: merge, moves: waits(24), gates: checks(5) })).toEqual([]); + expect(say({ cfg: merge, moves: waits(25), gates: checks(4) })).toEqual([]); + expect(say({ cfg: merge, moves: waits(25), gates: checks(5, "fail") })).toEqual([]); + // Nobody is on the gate now: nothing to take off. + expect(say({ moves: waits(25), gates: checks(5) })).toEqual([]); + }); +}); + +describe("retroText", () => { + test("every section, in plain lines", () => { + const cfg = cfgOf("claude:\n max_budget_usd: 2\n"); + const text = retroText(buildRetro(input({ + cfg, + moves: [ + mv("add-numbers", ago(6), "draft", "accepted"), + mv("add-numbers", ago(5), "merge-approved", "merged"), + ...blocks("budget", 1, "slow-one"), + mv("odd-one", ago(1), "merge-approved", "blocked", "The work for this change is missing. To build it again, set status to plan-approved."), + ], + gates: [gateRow("add-numbers", "spec", "spec-check", "fail"), gateRow("add-numbers", "spec", "spec-check", "pass")], + phases: [phase("add-numbers", "build", 1.5), phase("slow-one", "fix-1", 0.5), phase("_chat", "chat", 0.25)], + questions: [ev("add-numbers", "question", ago(6, 1), { question: "Which?" })], + requests: [ev("add-numbers", "request", ago(6, 3), { kind: "answer", result: "applied" })], + }))); + expect(text).toBe([ + "The last 7 days (since Oct 20).", + "", + "What happened", + "- Reached main: 1 (add-numbers).", + "- Finished: 0.", + "- Blocked: 2 (slow-one and odd-one).", + "- Closed: 0.", + "- From accepted to merged: 24 hours at the median, over 1 change.", + "", + "Where changes blocked", + "- Hit the spending limit: 1 (slow-one).", + "- Other: 1 (odd-one).", + " - odd-one: The work for this change is missing.", + "", + "Gates", + "- spec-check: ran 2 times; 0 of 1 change passed the first time.", + "", + "Spending", + "- $2.00 in all, over 2 changes ($1.00 a change).", + "- build: $1.50 (75%), $1.50 a change.", + "- fix: $0.50 (25%), $0.50 a change.", + "- Chat: $0.25, not counted above.", + "", + "Waiting for people", + "- Questions: 1 wait, 2 hours at the median.", + "", + "Suggestions", + "- 1 step hit the session spending limit of $2.00: raise claude.max_budget_usd with loopstra setup budgets.", + "", + ].join("\n")); + }); + + test("quiet sections say so, and no suggestions says why", () => { + const text = retroText(buildRetro(input({ days: 1, gates: [gateRow("a", "merge", "pr-approved", "pass")] }))); + expect(text).toBe([ + "The last day (since Oct 26).", + "", + "What happened", + "- Reached main: 0.", + "- Finished: 0.", + "- Blocked: 0.", + "- Closed: 0.", + "", + "Where changes blocked", + "- Nothing blocked.", + "", + "Gates", + "- No automatic checks ran.", + "", + "Spending", + "- Nothing was spent.", + "", + "Waiting for people", + "- Nobody was waited for.", + "", + "Suggestions", + "- No suggestions: nothing in the trace points at a setting to change.", + "", + ].join("\n")); + }); + + test("an empty window says nothing happened, then the suggestions", () => { + expect(retroText(buildRetro(input()))).toBe("Nothing happened in the last 7 days.\n\nSuggestions\n- No suggestions: nothing in the trace points at a setting to change.\n"); + expect(retroText(buildRetro(input({ days: 1 })))).toStartWith("Nothing happened in the last day.\n"); + }); + + test("long lists of changes are cut at five", () => { + const moves = ["a", "b", "c", "d", "e", "f", "g"].map((s) => mv(s, ago(1), "closed", "closed")).concat(["a", "b", "c", "d", "e", "f", "g"].map((s) => mv(s, ago(1), "reviewing", "closed"))); + expect(retroText(buildRetro(input({ moves })))).toContain("\n- Closed: 7 (a, b, c, d, e and 2 more).\n"); + }); +}); + +describe("loopstra retro on a trace", () => { + async function setUp(root: string): Promise<void> { + mkdirSync(join(root, "loopstra"), { recursive: true }); + await Bun.write(configPath(root), BASE); + } + + test("a brand-new trace, and one with only chat in it, say nothing happened", async () => { + const t = tempDir(); + try { + await setUp(t.path); + const empty = "Nothing happened in the last 7 days.\n\nSuggestions\n- No suggestions: nothing in the trace points at a setting to change.\n"; + expect(await renderRetro(t.path, 7)).toBe(empty); + const trace = Trace.open(t.path); + trace.phaseEnd("_chat", trace.phaseStart("_chat", "chat", "agent"), { status: "success", costUsd: 0.5 }); + trace.event("_chat", "question", { question: "?" }); + trace.close(); + expect(await renderRetro(t.path, 7)).toBe(empty); + } finally { t.cleanup(); } + }); + + test("reads the window's rows from the trace, leaving older ones out", async () => { + const t = tempDir(); + try { + await setUp(t.path); + const trace = Trace.open(t.path); + trace.statusChange("old", "draft", "accepted"); + trace.statusChange("add-numbers", "draft", "accepted"); + trace.statusChange("add-numbers", "merge-approved", "merged"); + trace.statusChange("old", "merge-approved", "merged"); + trace.gate("add-numbers", "spec", "spec-check", "pass", "ok"); + trace.phaseEnd("add-numbers", trace.phaseStart("add-numbers", "build", "agent"), { status: "success", costUsd: 1 }); + trace.close(); + const db = new Database(join(t.path, ".loopstra", "trace.db")); + const old = new Date(Date.now() - 30 * DAY).toISOString(); + db.run("UPDATE events SET ts = ? WHERE slug = 'old' AND payload LIKE '%merged%'", [old]); + db.close(); + const text = await renderRetro(t.path, 3); + expect(text).toStartWith("The last 3 days (since "); + expect(text).toContain("\n- Reached main: 1 (add-numbers).\n"); + expect(text).toContain("\n- spec-check: ran 1 time; 1 of 1 change passed the first time.\n"); + expect(text).toContain("\n- $1.00 in all, over 1 change ($1.00 a change).\n"); + } finally { t.cleanup(); } + }); +}); From c17c76cfc78d4d07b38b51d1e0588d50c31e5fc5 Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 21:59:37 -0500 Subject: [PATCH 09/21] feat(retro): document loopstra retro in the README, the skill and the decisions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- README.md | 3 +++ docs/decisions.md | 15 +++++++++++++++ templates/skill/SKILL.md | 2 ++ 3 files changed, 20 insertions(+) diff --git a/README.md b/README.md index 71d8dd6d..50b58bb2 100644 --- a/README.md +++ b/README.md @@ -164,8 +164,11 @@ loopstra status # loop state, what needs a person, then every change: pr loopstra tail # live events; `loopstra tail <slug>` for one change loopstra ui # the console at http://127.0.0.1:4646 (--port <n>), with the orchestrator (see Chat) loopstra judges # how often the cheaper judges agreed with the strong ones (speed.cheap_judges) +loopstra retro # how the loop did over the last 7 days, and what to change (--days <n>) ``` +`loopstra retro` reads the trace back and reports on a window of days (7 unless `--days` says otherwise): which changes reached main, finished, blocked or were closed, and how long a change took from accepted to merged; where changes blocked, grouped by why (timed out, tests kept failing, the reviewer kept finding problems, checks on GitHub failed, and so on); each gate check, how often it ran and how many changes passed it the first time; what the loop spent, by kind of step, with chat on its own line; and how long changes waited for a person at each review status and for answers to questions. It ends with suggestions, each naming its evidence and the setting to change: more fix loops or review rounds, a longer timeout, a higher session limit, trying the cheaper judges or letting them go first, or taking a person off a gate whose checks always pass while changes wait on them for days. It only reads; changing a setting is up to you. + The loop line reads Running, Paused, Stopping, Stopped, "Stopped — it did not shut down cleanly" (the loop process is gone: killed or crashed; start it again), or "Not responding" (the process is there but has not checked in: hung, or the machine slept). Under it, `status` prints a "Needs attention" block, or "Nothing needs you right now.": a pause, failing tests on main (before main has ever passed, that they have not passed yet), main out of step with GitHub, GitHub not answering about pull requests for 15 minutes while a change waits on its pull request (gh signed out, say, or a token GitHub no longer takes; until it answers again, or no change waits on one), a settings problem, blocked changes, unreadable intents, and changes waiting for a person (drafts, and reviews with a person on the gate). The console's inbox shows the same list, labelled "Blocked" or "Waiting for you". Long notes wrap to the terminal's width. diff --git a/docs/decisions.md b/docs/decisions.md index 14d28af6..7b1401e7 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -888,3 +888,18 @@ A new `agents` block in `loopstra/config.yaml` gives each step an agent to play, So these sessions get main's setup, read from the step's commit of main (`git ls-tree` and `cat-file`) into a temporary folder of the session's own, removed when it ends: its settings merged into `--settings` (lists and maps joined, so both Loopstra's hook and main's run), its CLAUDE.md files and `.claude/rules` (which an added dir loads too, checked live) as a read-only `--add-dir`, its `.claude/skills`, `agents` and `commands` as the plugin `project`, and its `.mcp.json` servers less those its settings disable. A first version read them from the step's snapshot, a checkout: the review of this change found that a build can write the snapshot with any command it may run (only the edit tools are denied there), which is why agent files were already read from the commit, and it made every branch step check out all of main. The limits: main's subfolder CLAUDE.md files, and files a CLAUDE.md imports from elsewhere in the repository, do not load in these sessions; a hook or MCP server of main's that runs a script from the project folder runs the branch's copy; and main's settings can still turn hooks off or allow more, as they always could. - **A session runs on the config its prompt was rendered from.** A phase read loopstra/config.yaml three times (prompt, model, agents), so a setting saved in between could give a session agents its prompt did not name. The step now remembers the config each recent prompt was rendered from, and the session that sends the prompt uses it, as do the keys that match a prompt to earlier work (the judge comparison, the plan written ahead, the early review); a retry renders and reads again. - **Shared helpers, once.** `readText` is in `fsutil.ts`, frontmatter parsing (`splitFrontmatter`) in `agent-files.ts`, the agent and skill name rule (`ITEM_NAME`) in `agent-library.ts`, `TEAM_NAME` and `once` in `agents.ts` (which the page can import), and the empty-map cleanup is `ConfigDocument.dropEmptyMaps`. + +## loopstra retro (2026-10-27) + +- **A read-only report over the trace, in plain text like `loopstra judges`.** `src/retro.ts` is pure: it builds the report from rows the trace gives (`statusChangesSince`, `gatesSince`, `phasesSince`, `eventsOfTypeSince`), the config and the judges' report, and `src/commands/retro.ts` reads them. No schema change and no new storage; it writes nothing. +- **`--days` is a whole number from 1 to 3650; 7 without it.** Anything else is the usage line, like a bad port. One day reads "the last day". The first day of the window is shown in local time ("since Oct 20"). +- **Every status move is read, the rest only the window's.** A change merged this week may have been accepted long before, so the median from accepted to merged needs its whole history. Its start is the move to accepted, or, for a change written as accepted, its first move. +- **A move to the same status is not a move.** It only rewrote the note, so it neither counts as a block (or a merge) nor ends a wait. +- **Blocks are told apart by the constants the runtime writes them from.** The notes that were local are exported (`TESTS_KEPT_FAILING_NOTE`, `REVIEW_STILL_FOUND_NOTE`, `UNMET_NOTE`, `SESSION_LIMIT_NOTE` and the rest), and a note's kind is the first whose constant it contains, so a reworded note cannot drift from its kind. A failed session's note (timed out, crashed, the spending limit, too many questions) is looked for first, since a stage's note may lead with its own words ("The finished change could not be checked. The assistant took too long..."). A project command that kept running past its limit after every fix counts with tests that kept failing (the fix loops ran out either way), and reviews used up after late fixes with the reviewer that kept finding problems. Any other note is listed by its first sentence, without the retry advice. +- **Gates leave out what a person did.** `done/human` and `merge/pr-approved` record a person, not a check, so they are not gate lines. A change passed a check the first time when its first row in the window passed. Lines follow the stages' order. +- **Spending counts ended phases only.** A running phase's cost is a budget hold, not spending. "A change" is a change with a phase that ended in the window; kinds of step that cost nothing (the code phases) are not listed. Judging is every spec-check, plan-challenge and review session, the cheaper judges' and the early review's included. +- **A wait at merge-review is a person's unless its note says it waits for GitHub's checks.** The other review statuses are entered only when a person decides, so every wait there is a person's. This reads what happened, whatever the config says now. A wait starts in the window and ends at the change's next move to another status; one still open counts until now and is reported as still waiting, since the longest waits are the ones that matter. A question's wait ends at an applied answer or at the change's next status move. +- **Suggestion details.** Timeouts are counted from blocks, not from timeouts retried away. The longer timeout is half as long again, rounded up to 5 minutes. A person may come off a gate only when the config has one on it now, the median wait there is over 24 hours, and every automatic check of that gate in the window passed, at least 5 times in all. Whether a cheaper judge has earned going first is `judgeReport`'s new `earned` field, not its words. +- **The two rules on settings from the other parts slot in later.** "Set `gates.merge.risky`" and "turn on `stages.build.acceptance_tests`" read keys added on other branches; `suggestions` marks where they go, and `RetroFacts` already carries what they read (`merged`, `blocks.unmet`). +- **An empty window is said in one line.** Nothing moved, no gate ran and no change's phase or question started in the window (chat alone does not count): "Nothing happened in the last 7 days.", then the suggestions, which need data and so say there are none. +- **Lists of changes stop at five**, then "and N more". diff --git a/templates/skill/SKILL.md b/templates/skill/SKILL.md index d6973276..10ca2d1a 100644 --- a/templates/skill/SKILL.md +++ b/templates/skill/SKILL.md @@ -62,6 +62,8 @@ If the loop line says "Paused — " followed by "The assistant is unavailable", Read the `note` in the change's `intent.md`. Explain the choices: follow the status the note names (it says which status to set to try again, or the two ways on); `resume_from`, the last approved state, is the fallback when the note names none; fix something first and then retry; or `closed`. Make the edit only when the person says which. For details, read `.loopstra/runs/<slug>/events.jsonl` or the phase folders under `.loopstra/runs/<slug>/phases/`. ## Tune +To see how the loop is doing before changing anything, run `loopstra retro` (the last 7 days; `--days 30` for longer). It says what reached main, where changes blocked and why, how each gate check did, what the loop spent by kind of step, and how long changes waited for a person, then suggests settings to change, each with its evidence. Explain its suggestions in plain words and change a setting only when the person agrees; retro itself changes nothing. + Edit `loopstra/config.yaml`, or a phase's prompt. Prompts ship with Loopstra: to change one, have them run `loopstra customize <phase>` (`loopstra customize` lists them), which copies it to `loopstra/prompts/<phase>.md` and records it in `loopstra/stamp.json`; edit that copy and commit both. Deleting the copy goes back to the shipped prompt. Changes take effect on the next pass; commit them. Keep prompts short and explicit; keep gate defaults deterministic. To change which MCP tools the loop may use, list them and ask again as in Onboard step 5. To change the agents (a step's agent, subagents, skills, teams), go through Onboard step 6 again, run `loopstra setup agents`, or use the Agents part of the console's Settings page. ## Upgrade From 6f59170486c868cbff87d3f809350d344d659ee1 Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:00:07 -0500 Subject: [PATCH 10/21] feat(acceptance): the build writes acceptance tests first and holds them When stages.build.acceptance_tests is on, a fresh acceptance session writes tests from Done when before the build session. The runtime commits them on the branch, records acceptance.md on main, and keeps the files in the run folder's acceptance marker, once per build. Every session gets them in LOOPSTRA_ACCEPTANCE; a refusal is named in the block note, and the drift check leaves them out. A failed session does not block: its files are dropped and the build goes on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- src/context.ts | 19 ++++ src/phases.ts | 29 ++++- src/stages/acceptance.ts | 97 +++++++++++++++++ src/stages/build.ts | 20 +++- tests/fake-claude/claude.ts | 5 +- tests/fake-claude/fixtures/acceptance.jsonl | 3 + tests/unit/stages-build.test.ts | 115 ++++++++++++++++++++ 7 files changed, 276 insertions(+), 12 deletions(-) create mode 100644 src/stages/acceptance.ts create mode 100644 tests/fake-claude/fixtures/acceptance.jsonl diff --git a/src/context.ts b/src/context.ts index f85e1ac2..dace4f56 100644 --- a/src/context.ts +++ b/src/context.ts @@ -616,6 +616,25 @@ export function clearSession(ctx: StepContext, key: string): void { writeMarker(ctx, SESSIONS, JSON.stringify(all, null, 2)); } +/** + * Run-folder marker: the acceptance test files this build's acceptance step wrote from Done when, as a + * JSON list of paths in the repository. Its being there means the step ran for this build; a new build + * clears it (see runBuildStep). + */ +export const ACCEPTANCE = "acceptance"; + +const AcceptanceList = z.array(z.string()); + +/** The acceptance test files the build's sessions may not edit (see ACCEPTANCE); none when there is no readable marker. */ +export function acceptanceFiles(ctx: StepContext): string[] { + try { + const list = AcceptanceList.safeParse(JSON.parse(readMarker(ctx, ACCEPTANCE) ?? "[]")); + return list.success ? list.data : []; + } catch { + return []; + } +} + /** Run-folder file: the note a person sent the change back with, for the next writing or build session (see sentBackNote). */ export const SENT_BACK = "sent-back.md"; diff --git a/src/phases.ts b/src/phases.ts index 41ddcc4e..f12585ad 100644 --- a/src/phases.ts +++ b/src/phases.ts @@ -9,7 +9,7 @@ import { BUILTIN_AGENTS, isEmptySetup, resolveSetup, setupLines, type AgentSetup import { findAgent, setupSession, userHome, type AgentDefinition, type Where } from "./agent-files"; import type { Trace } from "./trace"; import { GitTimeout } from "./git"; -import { CRASH_NOTE, TIMEOUT_NOTE, type StepContext } from "./context"; +import { acceptanceFiles, CRASH_NOTE, TIMEOUT_NOTE, type StepContext } from "./context"; import { Envelopes, jsonSchemaFor, type Envelope, type PhaseName } from "./envelopes"; import { promptPath, renderPrompt, type PromptVars } from "./prompts"; import { issuesText, tracedSession } from "./sessions"; @@ -534,7 +534,7 @@ async function attempt<N extends PhaseName>(ctx: StepContext, spec: AgentPhaseSp }, seq); } const dir = join(ctx.runDir, "phases", `${seq}-${traceName}`); - // The protect-tests hook writes here the test files it refused to let a fix session edit. + // The protect-tests hook writes here the test files it refused to let a session edit. const protectedLog = join(dir, "protected-tests.txt"); // The session's question, when it asked one (see agentPhase); `refused`: the note when it may not ask. let asked: Question | null = null; @@ -550,7 +550,7 @@ async function attempt<N extends PhaseName>(ctx: StepContext, spec: AgentPhaseSp allowedTools: toolsFor(ctx, spec.tools, setup), // With the setup's and main's: no edits in main's snapshot when the session may read it as an added dir. disallowedTools: [...new Set([...disallowedFor(spec.tools), ...(extra?.deny ?? []), ...(main?.deny ?? [])])], - // The test-protection hook, from the installed package; it acts only in a fix session. + // The test-protection hook, from the installed package; it acts only in sessions that write (see the hook). settings: hookSettings(), // The step's agent setup (see setupRun), which may also replace the settings. ...extra?.run, @@ -560,8 +560,10 @@ async function attempt<N extends PhaseName>(ctx: StepContext, spec: AgentPhaseSp maxBudgetUsd: capUsd, resume: spec.resume, // A session that may run the project's commands gets the change's slot, as its commands do. + // LOOPSTRA_ACCEPTANCE: the tests written from Done when, which the hook keeps the build's sessions from editing. env: { LOOPSTRA_PHASE: spec.name, LOOPSTRA_SLUG: ctx.slug, LOOPSTRA_BASE: ctx.cfg.main_branch, LOOPSTRA_PROTECTED_LOG: protectedLog, + LOOPSTRA_ACCEPTANCE: JSON.stringify(acceptanceFiles(ctx)), ...(RUNS_COMMANDS.has(spec.tools) ? slotEnv(ctx.slug) : {}), ...main?.env, ...(spec.env ?? {}), }, signal, @@ -642,10 +644,25 @@ function protectedTests(log: string): string[] { } } -/** Names the protected test files a failed fix wanted to edit, so a person can make the edit instead of a rebuild. */ +/** A path as the hook compares it: forward slashes, and without case where the file system ignores it. */ +function samePath(p: string): string { + const s = p.replace(/\\/g, "/"); + return process.platform === "win32" || process.platform === "darwin" ? s.toLowerCase() : s; +} + +/** + * Names the protected test files a failed session wanted to edit, so a person can make the edit instead + * of a rebuild: tests written from Done when (see ACCEPTANCE in context.ts), and tests a fix may not change. + */ function protectedNote(ctx: StepContext, files: string[]): string { - const list = files.map((f) => `\`${f}\``).join(", "); - return `It was not allowed to edit ${files.length === 1 ? "this test file" : "these test files"}, which this branch did not add or edit: ${list}. If the test is wrong, an engineer can fix it on the branch ${ctx.branch}.`; + const list = (fs: string[]) => fs.map((f) => `\`${f}\``).join(", "); + const accepted = new Set(acceptanceFiles(ctx).map(samePath)); + const fromDoneWhen = files.filter((f) => accepted.has(samePath(f))); + const others = files.filter((f) => !accepted.has(samePath(f))); + const said: string[] = []; + if (others.length) said.push(`It was not allowed to edit ${others.length === 1 ? "this test file" : "these test files"}, which this branch did not add or edit: ${list(others)}.`); + if (fromDoneWhen.length) said.push(`It was not allowed to edit ${fromDoneWhen.length === 1 ? "the acceptance test" : "the acceptance tests"} ${list(fromDoneWhen)}, written from Done when before the build.`); + return `${said.join(" ")} If the test is wrong, an engineer can fix it on the branch ${ctx.branch}.`; } const PROBE_SCHEMA = { type: "object", properties: { ok: { type: "boolean" } }, required: ["ok"], additionalProperties: false }; diff --git a/src/stages/acceptance.ts b/src/stages/acceptance.ts new file mode 100644 index 00000000..35ee458d --- /dev/null +++ b/src/stages/acceptance.ts @@ -0,0 +1,97 @@ +import { existsSync } from "node:fs"; +import { isAbsolute, join, relative } from "node:path"; +import { ACCEPTANCE, writeArtifact, writeMarker, type Failure, type StepContext } from "../context"; +import type { Envelope } from "../envelopes"; +import { passOn, type Git } from "../git"; +import { agentPhase } from "../phases"; +import { errorText } from "../shell"; +import { bullets, KEEP_UNFINISHED, saveWork, type Artifacts } from "./shared"; + +/** What acceptance.md and the done-check say when the change has no acceptance tests. */ +const NONE_WRITTEN = "No acceptance tests were written for this change."; + +/** The heading of acceptance.md's list of tests, which the done-check reads. */ +const COVERED = "Covered by tests"; + +/** + * The acceptance step (stages.build.acceptance_tests), run before the build session: a fresh session + * in the change's worktree writes tests from the owner's Done when list. The runtime commits them on + * the branch, records acceptance.md on main, and writes the ACCEPTANCE marker with the test files, which + * the build's sessions may not edit. A session that fails does not block the change: the trace says + * why, acceptance.md says no tests were written, what it wrote is dropped, and the build goes on as + * without the setting. Only a commit that fails returns a failure (the build's own would fail too). + */ +export async function writeAcceptance(ctx: StepContext, wt: Git, a: Artifacts): Promise<{ ok: true } | Failure> { + // What an earlier build left unsaved is kept first, so the acceptance commit holds only the tests. + const leftover = await saveWork(ctx, wt, KEEP_UNFINISHED); + if (!leftover.ok) return leftover; + const before = await wt.headSha(); + const r = await agentPhase(ctx, { + name: "acceptance", permissionMode: "acceptEdits", tools: "build", cwd: ctx.worktreeDir, + vars: { done_when: ctx.intent.file.sections["Done when"] ?? "", spec: a.spec, plan: a.plan, test_command: ctx.cfg.commands.test }, + skills: ctx.cfg.stages.build.skills, + }); + if (!r.ok) { + ctx.trace.event(ctx.slug, "error", { where: "acceptance", reason: r.reason, note: "the acceptance step did not finish; the build goes on without acceptance tests" }); + await dropUnsaved(ctx, wt); + await writeArtifact(ctx, "acceptance.md", acceptanceRecord(null, [])); + writeMarker(ctx, ACCEPTANCE, "[]"); + return { ok: true }; + } + const subject = `loopstra(${ctx.slug}): acceptance tests`; + const said = r.envelope.commit_message.trim(); + const saved = await saveWork(ctx, wt, said ? `${subject}\n\n${said}` : subject); + if (!saved.ok) return saved; + // What git says the session wrote, and the tests it names that are on the branch already (an + // earlier build's step wrote them, and this one kept them). + const written = (await wt.run(["diff", "--name-only", "-z", "--diff-filter=d", before, "HEAD", "--"])).out.split("\0").filter(Boolean); + const tests = r.envelope.tests.map((t) => ({ ...t, file: inRepo(ctx, t.file) })); + const named = tests.map((t) => t.file).filter((f) => f && existsSync(join(ctx.worktreeDir, f))); + const files = [...new Set([...written, ...named])]; + ctx.trace.event(ctx.slug, "command", { command: "acceptance", files, tests: tests.length, not_testable: r.envelope.not_testable.length }); + await writeArtifact(ctx, "acceptance.md", acceptanceRecord({ tests, not_testable: r.envelope.not_testable }, files)); + writeMarker(ctx, ACCEPTANCE, JSON.stringify(files)); + return { ok: true }; +} + +/** A path the session named, as a path in the repository with forward slashes ("" when it is outside the worktree). */ +function inRepo(ctx: StepContext, file: string): string { + let p = file.trim(); + if (isAbsolute(p)) p = relative(ctx.worktreeDir, p); + p = p.replace(/\\/g, "/"); + while (p.startsWith("./")) p = p.slice(2); + return p.startsWith("../") || isAbsolute(p) ? "" : p; +} + +/** Puts the worktree back as its last commit has it, after a session that failed; a failure is traced, and the build's commit may then include what it left. */ +async function dropUnsaved(ctx: StepContext, wt: Git): Promise<void> { + try { + await wt.putBack(await wt.changedTrackedFiles()); + await wt.run(["clean", "-f", "-d", "-q"]); + } catch (e) { + passOn(e); + ctx.trace.event(ctx.slug, "error", { where: "acceptance", error: errorText(e), note: "what the acceptance session wrote was not removed; the build's commit may include it" }); + } +} + +/** Ends a sentence with a period unless it has its own ending. */ +const sentence = (s: string) => (/[.!?]$/.test(s.trim()) ? s.trim() : `${s.trim()}.`); + +/** + * acceptance.md, for the owner: each test by the criterion it covers (the owner's words), then each + * criterion no test covers and why. `e` is null when the session did not finish; its test files are + * paths in the repository, and a test whose file is not among `files` is left out. + */ +function acceptanceRecord(e: Pick<Envelope<"acceptance">, "tests" | "not_testable"> | null, files: string[]): string { + const head = "# Acceptance tests\n\n"; + if (!e) return `${head}${NONE_WRITTEN} The step that writes them from Done when did not finish, so the build went on without them.\n`; + const held = new Set(files); + const covered = e.tests + .filter((t) => held.has(t.file)) + .map((t) => `${t.criterion.trim().replace(/\.$/, "")}: \`${t.file}\`, "${t.test}"`); + const open = e.not_testable.map((n) => `${n.criterion.trim().replace(/\.$/, "")}: ${sentence(n.why)}`); + const parts = [covered.length ? "Written from Done when before the build. The build may not change them." : `${NONE_WRITTEN} No criterion in Done when could be checked by a test.`]; + if (covered.length) parts.push(`## ${COVERED}\n${bullets(covered)}`); + if (open.length) parts.push(`## Not covered by a test\n${bullets(open)}`); + return `${head}${parts.join("\n\n")}\n`; +} diff --git a/src/stages/build.ts b/src/stages/build.ts index 5beadbed..81839bd5 100644 --- a/src/stages/build.ts +++ b/src/stages/build.ts @@ -1,7 +1,8 @@ import { diffWithinPlan, parsePlanFiles } from "../checks"; -import { block, clearMarker, readMarker, saveSession, SENT_BACK, setStatus, writeMarker, type Failure, type StepContext, type StepResult } from "../context"; +import { ACCEPTANCE, acceptanceFiles, block, clearMarker, readMarker, saveSession, SENT_BACK, setStatus, writeMarker, type Failure, type StepContext, type StepResult } from "../context"; import { Git } from "../git"; import { agentPhase, alongside, codePhase, mayOverlap, phasePrompt, projectCommand, type AgentPhaseSpec } from "../phases"; +import { writeAcceptance } from "./acceptance"; import { EARLY_REVIEW, keepEarlyReview, reviewAhead } from "./early-review"; import { errorText, type CommandResult } from "../shell"; import { StopRequested } from "../stop"; @@ -9,7 +10,11 @@ import { artifacts, bullets, buildSession, MERGING, PR_LEASE, PR_NUMBER, PR_PUSH /** Stage 3 build half plus Stage 4. Called for plan-approved and building. Ends at reviewing or blocked. */ export async function runBuildStep(ctx: StepContext): Promise<StepResult> { - if (ctx.intent.file.frontmatter.status !== "building") await setStatus(ctx, "building"); + if (ctx.intent.file.frontmatter.status !== "building") { + // A new build writes its own acceptance tests; one resumed after a stop keeps what it wrote. + clearMarker(ctx, ACCEPTANCE); + await setStatus(ctx, "building"); + } clearMarker(ctx, MERGING); // a new build is never an earlier merge in progress clearMarker(ctx, RECONCILED_PLAN); // nor is it compared with a plan reconciled for an earlier build clearMarker(ctx, EARLY_REVIEW); // nor reviewed by a review of an earlier build @@ -39,6 +44,12 @@ export async function runBuildStep(ctx: StepContext): Promise<StepResult> { const a = await artifacts(ctx); const stage = ctx.cfg.stages.build; + // Tests from the owner's Done when list, written before the build, once per build. + if (stage.acceptance_tests && readMarker(ctx, ACCEPTANCE) === null) { + const wrote = await writeAcceptance(ctx, wt, a); + if (!wrote.ok) return block(ctx, wrote.note, wrote); + } + const feedback = readMarker(ctx, SENT_BACK)?.trim() ?? ""; const phase: AgentPhaseSpec & { name: "build" } = { name: "build", permissionMode: "acceptEdits", tools: "build", cwd: ctx.worktreeDir, @@ -54,10 +65,11 @@ export async function runBuildStep(ctx: StepContext): Promise<StepResult> { // Used: the build it asked to change is saved. clearMarker(ctx, SENT_BACK); - // Plan drift: files changed that main's plan did not list. + // Plan drift: files changed that main's plan did not list. The acceptance tests are not the build's to list. let plan = a.plan; + const accepted = new Set(acceptanceFiles(ctx)); const drift = await codePhase(ctx, "drift", async () => { - const changed = await wt.changedFilesSince(ctx.cfg.main_branch); + const changed = (await wt.changedFilesSince(ctx.cfg.main_branch)).filter((f) => !accepted.has(f)); const planned = parsePlanFiles(plan); if (planned === null) { // No file list to compare against: say so in the trace rather than flag every file. diff --git a/tests/fake-claude/claude.ts b/tests/fake-claude/claude.ts index f58ede84..59ccf43e 100644 --- a/tests/fake-claude/claude.ts +++ b/tests/fake-claude/claude.ts @@ -19,7 +19,7 @@ // $LOOPSTRA_FAKE_CALLS, when set, gets one line of args per call. // A sequence: $LOOPSTRA_FAKE_FIXTURE_DIR/<phase>-<n>.jsonl is used for the n-th call of that phase // (counted in <dir>/.count-<phase>), before <phase>.jsonl. $LOOPSTRA_FAKE_PROMPTS, when set, gets -// each prompt as one JSON line ({phase, args, prompt, cwd}). +// each prompt as one JSON line ({phase, args, prompt, cwd, acceptance}: acceptance is $LOOPSTRA_ACCEPTANCE). // One model (the --model argument): before everything above, $LOOPSTRA_FAKE_FIXTURE_DIR/<phase>@<model>-<n>.jsonl // (counted per model in <dir>/.count-<phase>@<model>), then <phase>@<model>.jsonl. Two sessions of one phase // that start together race on the plain sequence's counter, so a test that runs both judges uses these. @@ -131,13 +131,14 @@ if (process.env.LOOPSTRA_FAKE_ARGS) { LOOPSTRA_BASE: process.env.LOOPSTRA_BASE ?? null, LOOPSTRA_SLOT: process.env.LOOPSTRA_SLOT ?? null, LOOPSTRA_PROTECTED_LOG: process.env.LOOPSTRA_PROTECTED_LOG ?? null, + LOOPSTRA_ACCEPTANCE: process.env.LOOPSTRA_ACCEPTANCE ?? null, GIT_OPTIONAL_LOCKS: process.env.GIT_OPTIONAL_LOCKS ?? null, CLAUDE_CODE_ADDITIONAL_DIRECTORIES_CLAUDE_MD: process.env.CLAUDE_CODE_ADDITIONAL_DIRECTORIES_CLAUDE_MD ?? null, } })); } if (process.env.LOOPSTRA_FAKE_CALLS) appendFileSync(process.env.LOOPSTRA_FAKE_CALLS, JSON.stringify(args) + "\n"); -if (process.env.LOOPSTRA_FAKE_PROMPTS) appendFileSync(process.env.LOOPSTRA_FAKE_PROMPTS, JSON.stringify({ phase, args, prompt, cwd: process.cwd() }) + "\n"); +if (process.env.LOOPSTRA_FAKE_PROMPTS) appendFileSync(process.env.LOOPSTRA_FAKE_PROMPTS, JSON.stringify({ phase, args, prompt, cwd: process.cwd(), acceptance: process.env.LOOPSTRA_ACCEPTANCE ?? null }) + "\n"); /** Counts one more call of `name` in <dir>/.count-<name> and gives the count. */ async function nextCall(dir: string, name: string): Promise<number> { diff --git a/tests/fake-claude/fixtures/acceptance.jsonl b/tests/fake-claude/fixtures/acceptance.jsonl new file mode 100644 index 00000000..eddedd4d --- /dev/null +++ b/tests/fake-claude/fixtures/acceptance.jsonl @@ -0,0 +1,3 @@ +{"type":"system","subtype":"init","session_id":"fake-acceptance","cwd":"/","tools":[],"model":"fake"} +{"type":"fake_action","write":{"path":"tests/accept.test.ts","content":"import { expect, test } from \"bun:test\";\nimport { add } from \"../src/add\";\ntest(\"adds one and two\", () => { expect(add(1, 2)).toBe(3); });\n"}} +{"type":"result","subtype":"success","is_error":false,"session_id":"fake-acceptance","total_cost_usd":0.05,"usage":{},"structured_output":{"status":"success","summary":"one acceptance test; one criterion needs a person","commit_message":"test: acceptance tests for add","tests":[{"criterion":"add(1, 2) returns 3.","file":"tests/accept.test.ts","test":"adds one and two"}],"not_testable":[{"criterion":"People find add easy to use.","why":"needs a person"}]},"result":"ok"} diff --git a/tests/unit/stages-build.test.ts b/tests/unit/stages-build.test.ts index e6a80ef5..009c3427 100644 --- a/tests/unit/stages-build.test.ts +++ b/tests/unit/stages-build.test.ts @@ -699,3 +699,118 @@ describe("the first review alongside the verifier", () => { }); } }); + +describe("acceptance tests from Done when", () => { + const ON = "stages:\n build:\n acceptance_tests: true\n"; + type Call = { phase: string; prompt: string; acceptance: string | null }; + const calls = async (log: string): Promise<Call[]> => (await Bun.file(log).text()).trim().split("\n").map((l) => JSON.parse(l) as Call); + const wtOf = (repo: { path: string }) => join(repo.path, ".loopstra", "worktrees", "add-numbers"); + const ACCEPTANCE_MD = "# Acceptance tests\n\nWritten from Done when before the build. The build may not change them.\n\n## Covered by tests\n- add(1, 2) returns 3: `tests/accept.test.ts`, \"adds one and two\"\n\n## Not covered by a test\n- People find add easy to use: needs a person.\n"; + + test("off (the default), no acceptance session runs and nothing is held back", async () => { + const { repo, ctx, trace } = await planned({ test: "bun test" }); + try { + const log = join(repo.path, "prompts.jsonl"); + await withEnv({ LOOPSTRA_FAKE_PROMPTS: log }, () => runBuildStep(ctx)); + expect((await calls(log)).map((c) => c.phase)).not.toContain("acceptance"); + expect(phaseNames(trace)).not.toContain("acceptance"); + expect(existsSync(join(repo.path, "intent", "add-numbers", "acceptance.md"))).toBe(false); + expect((await calls(log)).find((c) => c.phase === "build")?.acceptance).toBe("[]"); + } finally { trace.close(); repo.cleanup(); } + }); + + test("on, a session writes tests before the build; they are committed on the branch, recorded on main, held from the build, and left out of the drift check", async () => { + const { repo, ctx, trace } = await planned({ test: "bun test" }, ON); + try { + const log = join(repo.path, "prompts.jsonl"); + const r = await withEnv({ LOOPSTRA_FAKE_PROMPTS: log }, () => runBuildStep(ctx)); + expect(r.ok).toBe(true); + expect((await readIntent(repo.path, "add-numbers")).file.frontmatter.status).toBe("reviewing"); + // Before the build; the test file is not in the plan, yet no reconcile runs. + expect(withoutEarlyReview(trace)).toEqual(["branch", "acceptance", "build", "drift", "test-1", "verify"]); + const seen = await calls(log); + expect(seen.map((c) => c.phase).slice(0, 2)).toEqual(["acceptance", "build"]); + const asked = seen[0]!; + expect(asked.prompt).toContain("- add(1, 2) returns 3."); + expect(asked.prompt).toContain("# Plan: add"); + expect(asked.prompt).toContain("`bun test`"); + // Every session after it is given the files it wrote. + for (const c of seen.slice(1)) expect({ phase: c.phase, acceptance: c.acceptance }).toEqual({ phase: c.phase, acceptance: JSON.stringify(["tests/accept.test.ts"]) }); + // Its own commit on the branch, before the build's. + const wt = new Git(wtOf(repo)); + const subjects = (await wt.run(["log", "--format=%s", "main..HEAD"])).out.trim().split("\n").reverse(); + expect(subjects.slice(0, 2)).toEqual(["loopstra(add-numbers): acceptance tests", "feat: add function"]); + expect((await wt.run(["show", "--name-only", "--format=", "HEAD~1"])).out.trim()).toBe("tests/accept.test.ts"); + // Recorded on main for the owner, and in the run folder for the sessions that follow. + expect(await Bun.file(join(repo.path, "intent", "add-numbers", "acceptance.md")).text()).toBe(ACCEPTANCE_MD); + expect((await new Git(repo.path).run(["log", "-1", "--format=%s", "--", "intent/add-numbers/acceptance.md"])).out.trim()).not.toBe(""); + expect(readMarker(ctx, "acceptance")).toBe(JSON.stringify(["tests/accept.test.ts"])); + } finally { trace.close(); repo.cleanup(); } + }); + + test("a build resumed after a stop does not write its acceptance tests again; a new build does", async () => { + const { repo, ctx, trace } = await planned({ test: "bun test" }, ON); + try { + const log = join(repo.path, "prompts.jsonl"); + await withEnv({ LOOPSTRA_FAKE_PROMPTS: log }, () => runBuildStep(ctx)); + // Resumed: the status is still building and the run folder has the marker. + await writeIntentOnDisk(ctx.intent, { status: "building" }); + await new Git(repo.path).commitAll("resume"); + await ctx.reload(); + await withEnv({ LOOPSTRA_FAKE_PROMPTS: log }, () => runBuildStep(ctx)); + expect((await calls(log)).filter((c) => c.phase === "acceptance").length).toBe(1); + expect((await calls(log)).filter((c) => c.phase === "build").map((c) => c.acceptance)).toEqual([JSON.stringify(["tests/accept.test.ts"]), JSON.stringify(["tests/accept.test.ts"])]); + // A new build (from plan-approved) writes them again. + await writeIntentOnDisk(ctx.intent, { status: "plan-approved" }); + await new Git(repo.path).commitAll("rebuild"); + await ctx.reload(); + await withEnv({ LOOPSTRA_FAKE_PROMPTS: log }, () => runBuildStep(ctx)); + expect((await calls(log)).filter((c) => c.phase === "acceptance").length).toBe(2); + // The tests are on the branch already and unchanged, but the session names them, so they stay held. + expect(readMarker(ctx, "acceptance")).toBe(JSON.stringify(["tests/accept.test.ts"])); + expect((await readIntent(repo.path, "add-numbers")).file.frontmatter.status).toBe("reviewing"); + } finally { trace.close(); repo.cleanup(); } + }); + + test("an acceptance session that fails does not block: the trace says why, acceptance.md says none were written, its files are dropped and the build goes on", async () => { + const failing = [ + `{"type":"system","subtype":"init","session_id":"fake-acceptance","cwd":"/","tools":[],"model":"fake"}`, + `{"type":"fake_action","write":{"path":"tests/half.test.ts","content":"half\\n"}}`, + `{"type":"result","subtype":"success","is_error":false,"session_id":"fake-acceptance","total_cost_usd":0.01,"usage":{},"structured_output":{"status":"fail","summary":"could not find the test folder","commit_message":"","tests":[],"not_testable":[]},"result":"ok"}`, + ].join("\n") + "\n"; + const dir = fixtureDir({ acceptance: failing }); + const { repo, ctx, trace } = await planned({ test: "bun test" }, ON); + try { + const log = join(repo.path, "prompts.jsonl"); + const r = await withEnv({ LOOPSTRA_FAKE_FIXTURE_DIR: dir.path, LOOPSTRA_FAKE_PROMPTS: log }, () => runBuildStep(ctx)); + expect(r.ok).toBe(true); + expect((await readIntent(repo.path, "add-numbers")).file.frontmatter.status).toBe("reviewing"); + expect(trace.events("add-numbers").some((e) => e.type === "error" && e.payload.includes("acceptance") && e.payload.includes("agent-fail"))).toBe(true); + const md = await Bun.file(join(repo.path, "intent", "add-numbers", "acceptance.md")).text(); + expect(md).toStartWith("# Acceptance tests\n\nNo acceptance tests were written for this change."); + expect(md).not.toContain("## Covered by tests"); + expect(existsSync(join(wtOf(repo), "tests", "half.test.ts"))).toBe(false); + expect((await new Git(wtOf(repo)).run(["log", "--format=%s", "main..HEAD"])).out).not.toContain("acceptance tests"); + expect((await calls(log)).find((c) => c.phase === "build")?.acceptance).toBe("[]"); + // Not tried again when the build resumes. + expect(readMarker(ctx, "acceptance")).toBe("[]"); + } finally { trace.close(); repo.cleanup(); dir.cleanup(); } + }); + + test("a fix refused an acceptance test blocks with a note that names it and says how to change it", async () => { + const refused = [ + `{"type":"system","subtype":"init","session_id":"fake-fix-refused","cwd":"/","tools":[],"model":"fake"}`, + `{"type":"fake_action","refuse":"tests/accept.test.ts"}`, + `{"type":"result","subtype":"success","is_error":false,"session_id":"fake-fix-refused","total_cost_usd":0.01,"usage":{},"structured_output":{"status":"fail","summary":"The only fix is in the acceptance test","commit_message":""},"result":"ok"}`, + ].join("\n") + "\n"; + const dir = fixtureDir({ fix: refused }); + const { repo, ctx, trace } = await planned({ test: "exit 1" }, `${ON} max_fix_loops: 1\n`); + try { + const r = await withEnv({ LOOPSTRA_FAKE_FIXTURE_DIR: dir.path }, () => runBuildStep(ctx)); + expect(r.ok).toBe(false); + const i = await readIntent(repo.path, "add-numbers"); + expect(i.file.frontmatter.status).toBe("blocked"); + expect(i.file.frontmatter.note).toBe("The assistant reported it could not finish this step. It was not allowed to edit the acceptance test `tests/accept.test.ts`, written from Done when before the build. If the test is wrong, an engineer can fix it on the branch intent/add-numbers. When that is sorted out, set status to plan-approved to try again."); + } finally { trace.close(); repo.cleanup(); dir.cleanup(); } + }); +}); From 87f46fa626fe1a255467912e9da84d9f39e8943f Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:00:23 -0500 Subject: [PATCH 11/21] feat(risk): setup asks for risky files; docs for risky changes With nobody on the merge gate, loopstra setup's GitHub section asks which files need a person before merging, suggesting .github/**, **/migrations/** and the dependency and build files at the top of the repository; the file's own list is kept as the suggestion, - empties it, and --defaults never adds one. The README covers gates.merge.risky under GitHub and Configuration and merge_gate under When something needs a person; intent/README.md tells owners what merge_gate means; the config template shows a commented example; decisions.md records each call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- README.md | 17 ++++++++++++++++- docs/architecture.md | 1 + docs/decisions.md | 14 ++++++++++++++ src/setup/sections/github.ts | 28 +++++++++++++++++++++++++++- templates/config.yaml | 5 +++++ templates/intent-README.md | 2 ++ tests/unit/setup-github.test.ts | 28 ++++++++++++++++++++++++++++ 7 files changed, 93 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 71d8dd6d..1f79b9e6 100644 --- a/README.md +++ b/README.md @@ -240,6 +240,7 @@ The `status` and `note` lines at the top of `intent.md` say what to do; `intent/ - `blocked`: read the `note`. It says in plain words what went wrong and which status to set to try again (for example back to `plan-approved`), or you can set `closed`. A passing hiccup (the assistant crashed or took too long) says "To try again, set status to ..."; anything else says "When that is sorted out, ...". Loops are bounded (test-fix 3, review-revise 2). A step that hits its spending limit (`claude.max_budget_usd`, if set) blocks, and its note says an engineer can raise or remove it with `loopstra setup budgets`; so does a step that, on two days at the same status, used (nearly) the whole of the loop's daily budget itself and still ran out partway. - `spec-review`, `plan-review`: the automatic checks passed and a gate is set to wait for a person. Read `spec.md` or `plan.md`, then set `spec-approved` or `plan-approved`, or say what is wrong in the note and set the earlier status. Stepping a review status never advances it. - `merge-review`: checks passed and the change is waiting to go into the main code. Follow the note: with nobody on the merge gate it waits for the checks on GitHub (nothing to do; it merges once they pass, even while another change is being worked on, within about `poll_seconds`); with `pr`, approve the pull request on GitHub; with `status`, read `review.md` and set `merge-approved`. +- `merge_gate: pr` or `merge_gate: status` beside `merge-review`: the change is risky (it touches files named under `gates.merge.risky`, or is larger than its `lines`), so it waits for a person even with nobody on the merge gate. Its note names up to three of the files. With `pr`, approve the pull request on GitHub (without a remote, there is none: set `merge-approved`); with `status`, read `review.md` and set `merge-approved`. Loopstra writes the line, and removes it, each time the change passes its merge checks; deleting it by hand takes the change back to the config's merge gate. - `merged`: the change is in the main code. - `verifying`: with a person on the done gate, look at the result and set `done`. - `question: open`: an agent asked you something, and the change waits for your answer, whatever its status. See Steering. @@ -255,6 +256,20 @@ With a remote, the approving review pushes the change's branch and opens a pull - `pr`: also needs an approval on the PR (or a person setting `merge-approved`). When a branch rule requires reviews, GitHub decides whether the PR is approved. Otherwise Loopstra reads the reviews itself, and only those of the repository's owner, members and collaborators count (anybody can review on a public repository): one of them approving is enough, as long as none of their latest reviews asks for changes. That is broader than GitHub's own rule, which counts only people with write access, so a collaborator with read access counts here; for GitHub's rule, add a branch rule that requires approving reviews. - `status`: a person sets `merge-approved`, and the PR's checks must pass. +**Risky changes.** Some changes should not go in unseen even when the rest merge on their own: changes to CI, dependencies or migrations. `gates.merge.risky` names them: + +```yaml +gates: + merge: + human: none + risky: + paths: [".github/**", "package.json", "bun.lock", "**/migrations/**"] + lines: 1500 # optional: more lines added and removed than this is risky too + human: pr # or status; pr is the default +``` + +A change is risky when a file it changes (outside `intent/`, new or old name of a renamed file) matches one of `paths`, or when its lines added plus removed, outside `intent/` and not counting binary files, are more than `lines`. Patterns are globs on repository paths with `/`: `package.json` matches only at the top, `**/package.json` anywhere. With nobody on the merge gate, a risky change waits in `merge-review` for `risky.human` (`pr`: an approval on GitHub, or `merge-approved`; without a remote, `merge-approved`), and Loopstra writes `merge_gate:` in its `intent.md` (see When something needs a person). When the config already has a person on the merge gate, that person decides as before, and only the trace says the change was risky. The decision is made each time the change passes its merge checks, on the code that will merge, so a fix or a rebuild is scored again; commits a person pushes to the pull request are not. The trace has it as the merge gate's `risk` check: `pass` ("nothing risky"), or `waiting` with the files that matched and the size. Nothing is risky until `paths` or `lines` is set; `loopstra setup github` asks for `paths` when nobody is on the merge gate, suggesting `.github/**`, `**/migrations/**` and the dependency and build files at the top of the repository (`--defaults` never adds a list). + Merges go through `gh pr merge`, never twice, and only at the commit whose checks were read (`--match-head-commit`): commits pushed to the PR after that are taken into the branch here and merge once their own checks pass, and so does a PR someone updated on GitHub with "Update with rebase". A PR that is merged or closed on GitHub is noticed in any mode; a draft waits until it is marked ready for review. A PR someone merged by hand while a later review round committed fixes blocks, naming the branch, rather than record the fixes as merged. Loopstra knows each build's own PR by its number: a PR a person closes stays closed (the change blocks, also when it was being reviewed again), and a rebuild opens a new PR rather than taking an earlier build's merged or closed one for its own. Loopstra waits while any check is still running, and blocks when any fails or is cancelled. A PR with no checks at all merges; if the branch has workflows under `.github/workflows/`, it first waits up to 5 minutes after the PR opens or Loopstra pushes to it, for GitHub to start them. Pull requests are watched while other changes are worked on: each tick looks at them as its steps start, again every `poll_seconds` while any step runs, and once more when the last has ended, so a PR whose checks pass during another change's long build merges then, not after it. Just before `gh pr merge`, at every `speed.max_parallel_changes`, one included (the watcher may merge one PR while the lane works on the next change), Loopstra checks that the PR still has everything on `main`: a PR another merge left out of date is brought up to date, pushed, and waits for GitHub's checks again, so two changes never reach `main` together untested. After a merge, Loopstra deletes the branch on GitHub and keeps the local one until `main` has the change. The change is recorded as merged, and its done-check runs, only once local `main` has the PR's merge commit; until the sync can bring it in (for example while the main checkout has unsaved changes), the change waits, and the console and `loopstra status` say that its PR merged and why `main` here does not have it yet (the `main_sync` signal's reason). They also say when a PR waits for GitHub to merge it (a merge queue), or is ready but waits for `main` here to take what GitHub merged meanwhile. Without a remote the same checks run and Loopstra merges locally. Each tick, and right after a PR merges, Loopstra fetches and brings local `main` up to date with the remote's: commits only local `main` has are replayed on top of the remote's `main` in a throwaway copy (`.loopstra/sync`), and then the main checkout moves to the result in one step. The main checkout is never left in the middle of a rebase, even if Loopstra is killed; a move that a kill cuts short is put back as it was before the next tick does anything else. It shares `main` once, at the end of the tick (and just before it pushes a change's branch for its pull request), and only when every unpushed commit is its own (bookkeeping under `intent/`). Your own unpushed commits are never pushed for you; it waits until you push them. In `intent/`, the remote's version wins. @@ -437,7 +452,7 @@ The full design is in `docs/superpowers/specs/2026-09-28-loopstra-design.md`, wi - `commands`: test (required), install, lint, build, run - `claude`: models (`cheap`, `default`, `strong`, `max`: by default the aliases `haiku`, `sonnet`, `opus` and `fable`, which Claude Code resolves to the newest model of each family; a full name such as `claude-opus-5-5` pins one), `timeout_minutes`, optional spending limits (`max_budget_usd` per session, `max_budget_usd_per_day` for the loop's sessions together; unset means no limit), and `allowed_tools` for build sessions (the configured commands are always added; anything else a build needs, like `"Bash(make *)"`, goes here). Sessions load the MCP servers, skills and CLAUDE.md files of the machine and repository they run in (in a change's branch, `main`'s: see Where sessions read and write), but without an allow rule an MCP call is refused (a session cannot ask). A server can still be connecting when a session starts, so its tools may only appear after the first turn - `agents`: which agents, subagents, teams, skills, tools and MCP tools each step's session gets (see Agents for each step) -- `gates`: spec, plan, merge, done, each with `human` (`status` or `none`; merge also `pr`) and `agent` (independent reviewer); merge also `method` (`squash` or `merge`) +- `gates`: spec, plan, merge, done, each with `human` (`status` or `none`; merge also `pr`) and `agent` (independent reviewer); merge also `method` (`squash` or `merge`) and `risky` (`paths`, `lines`, `human`: changes that wait for a person whatever `human` says; see GitHub) - `stages`: per-stage model, `steps` (a model for single steps; see Models for each step), skills, `before`/`after` commands, and loop limits - `speed`: work that runs at the same time to save waiting; only `cheap_judges: first` changes who decides (once earned, a cheaper judge's pass stands). `overlap_judges` (on) runs the first review alongside the verifier (in a copy of the tested commit), and lessons alongside the final check; when the verifier fails, the review running beside it is stopped at once and round 1 runs after the fix. With `claude.max_budget_usd_per_day` set, they overlap only when what is left of the day covers two whole sessions (`claude.max_budget_usd`), so never without a session limit. `parallel_checks` (off) starts test, lint and build together, and stops the rest once one fails; leave it off if they write the same folder. `plan_ahead` (on) writes the first plan while the spec is being checked. The plan is stopped if the spec does not pass, and it is used only if the code on `main` it read (outside `intent/` and `loopstra/config.yaml`) is still the same commit's code when planning starts. It follows the same daily budget rule, and does not run with a person on the spec gate, without a spec judge, with plan `before` commands, or while another change is about to merge. `cheap_judges` (off) asks a cheaper model (`claude.models.default`) the same question as the spec check, the plan challenge and the review: `shadow` only records whether it agreed with the strong judge, which still decides, and `first` lets it pass something alone once it has also caught the strong judge's last 30 rejections, sending anything it rejects or is unsure of to the strong judge (`loopstra judges` shows the record). In `first`, every 5th call still runs both judges as an audit, with the strong judge deciding, so a later miss sends it back to `shadow`. Like the overlaps, `shadow` and the audits run only with room for every session at once (three when the early review runs beside the verifier, or the spec check beside a plan written ahead), so never with a daily budget and no session limit; then the strong judge decides alone, and in `first` the cheaper judge never decides alone. `think_ahead` (on) runs design and plan steps beside the changes running a step (see How it works). `max_parallel_changes` (1, up to 4) is how many changes may each run a step at the same time. Raise it only if the project's commands (test, lint, build, install) can run twice at once on this machine: no fixed ports, no shared test database, no output outside the worktree. Each command, and each session that may run commands, gets `LOOPSTRA_SLUG` and `LOOPSTRA_SLOT`: a small number (0, 1, ...) that no other change running at the same time has, so it can pick its own ports, database and Docker project names (port `3100 + LOOPSTRA_SLOT`, say). Main's health check runs alone and gets no slot. More changes at once also spend the daily budget and the subscription's usage limits faster. With `claude.max_budget_usd_per_day` set, only as many run at once as what is left of the day covers whole sessions (`claude.max_budget_usd`), so one at a time without a session limit; each overlap above also counts the sessions of the other changes running. - `signals`: how often main's health check runs diff --git a/docs/architecture.md b/docs/architecture.md index 9c0aa1b7..47de7b7a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -22,6 +22,7 @@ About 165 modules under `src/`, in eight layers. A module imports only from its - `heartbeat.ts` holds the loop's lock and pause. - `requests.ts`, `inbox.ts`, `conversation.ts` and `questions.ts` are what people leave for the loop. - `stamp.ts` holds what an upgrade knows. + - `risk.ts` scores a change's diff against `gates.merge.risky` (`scoreRisk`), for the merge gate. - `agents.ts` resolves a step's agent setup from the config (`resolveSetup`) and holds its rules: which steps are read-only, which git a step may never get, and which steps run as the build's agent. 4. **Records.** - `record-draft.ts` holds `RecordDraft`. diff --git a/docs/decisions.md b/docs/decisions.md index 14d28af6..d6a3c66e 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -888,3 +888,17 @@ A new `agents` block in `loopstra/config.yaml` gives each step an agent to play, So these sessions get main's setup, read from the step's commit of main (`git ls-tree` and `cat-file`) into a temporary folder of the session's own, removed when it ends: its settings merged into `--settings` (lists and maps joined, so both Loopstra's hook and main's run), its CLAUDE.md files and `.claude/rules` (which an added dir loads too, checked live) as a read-only `--add-dir`, its `.claude/skills`, `agents` and `commands` as the plugin `project`, and its `.mcp.json` servers less those its settings disable. A first version read them from the step's snapshot, a checkout: the review of this change found that a build can write the snapshot with any command it may run (only the edit tools are denied there), which is why agent files were already read from the commit, and it made every branch step check out all of main. The limits: main's subfolder CLAUDE.md files, and files a CLAUDE.md imports from elsewhere in the repository, do not load in these sessions; a hook or MCP server of main's that runs a script from the project folder runs the branch's copy; and main's settings can still turn hooks off or allow more, as they always could. - **A session runs on the config its prompt was rendered from.** A phase read loopstra/config.yaml three times (prompt, model, agents), so a setting saved in between could give a session agents its prompt did not name. The step now remembers the config each recent prompt was rendered from, and the session that sends the prompt uses it, as do the keys that match a prompt to earlier work (the judge comparison, the plan written ahead, the early review); a retry renders and reads again. - **Shared helpers, once.** `readText` is in `fsutil.ts`, frontmatter parsing (`splitFrontmatter`) in `agent-files.ts`, the agent and skill name rule (`ITEM_NAME`) in `agent-library.ts`, `TEAM_NAME` and `once` in `agents.ts` (which the page can import), and the empty-map cleanup is `ConfigDocument.dropEmptyMaps`. + +## Risky changes need a person (2026-10-27) + +- **Risk is facts about the diff, never an agent's judgment.** A change is risky when a file it changes matches a pattern in `gates.merge.risky.paths`, or when its lines added plus removed are more than `gates.merge.risky.lines`. Both leave out `intent/`, and a binary file adds no lines. It is predictable and testable, and costs no session. Nothing is risky until `paths` or `lines` is set, the default, so an upgraded loop behaves as before. +- **Patterns are Bun.Glob on repository paths with `/`.** A bare name (`package.json`) matches only at the top; `**/package.json` matches anywhere. A blank pattern is a config problem. +- **The diff is the branch against main from where they parted** (`git diff --numstat -z -M main...HEAD` in the change's worktree): only the change's own work counts, not what main gained since. `-z` keeps names with spaces and other alphabets as they are. A rename is one record, and it counts under its new name or its old one, whichever a pattern matches: moving a workflow out of `.github/` changes CI as much as moving one in. A diff git cannot make fails the step, so nothing merges unscored. +- **The decision is made at one place, `passMergeGate`, after the merge checks pass.** Every change Loopstra makes to the branch (build, fix, revise, the merge checks' own fixes) goes through another review and then through this point, so the decision is always about the code that will merge. Commits a person pushes to the pull request on GitHub are not scored again: a person is already involved. +- **`merge_gate` records who decides, in the same record as the move to `merge-review`.** The pull request watcher reads the change's frontmatter fresh on every look, so a status without the field could merge an unapproved risky change. `setStatus` takes frontmatter fields to set or remove in its one record for this. The field is written only when it changes who decides, that is with `gates.merge.human: none`; a change that is not risky loses one an earlier pass wrote, in that record or in the record of its merge. It is runtime-managed, and left out when blank, like `hold`. +- **One helper, `changeGates`, gives a change's own gates.** The config's, with `merge_gate` on the merge gate when the config has nobody there; it never takes a person off. `waitsForPerson`, `isRunnable` and `offers` apply it themselves, so the scheduler, `queue.md`, `status`, the console and chat need no change of their own; apply's approve and send-back check, the attention note, plan-ahead's "may merge first" and the merge stage (through `mergeHuman`) use it too. A person who deletes the line takes the change back to the config's gate: a person's edits win, as everywhere. +- **Without a remote, `pr` acts as `status`.** There is no pull request to approve, so a risky change waits in `merge-review` for `merge-approved` rather than blocking. The field still says `pr`, what the config asked for, so the change waits for an approval on GitHub if a remote is added before it merges. +- **With a person on the config's merge gate, only the trace changes.** The risk row is written, the field is not, and the note is the gate's usual one. +- **The note names the files.** Whoever approves a merge needs to know what to look at, so the owner's note names up to three of them ("and N more" after that), and says how to let the change in. A change risky only by size says how many lines changed against the limit. The trace's `merge`/`risk` row names up to ten, with the size when that tripped it: `pass` with "nothing risky", or `waiting`. No row is written when nothing is set to be risky. +- **Setup asks for paths only when nobody is on the merge gate,** in the GitHub section, where the merge gate is asked. The file's own list is the suggestion; without one, `.github/**`, `**/migrations/**`, and each common dependency or build file found at the top of the repository. `-` empties the list. `--defaults` never adds one: nothing is risky until a person says so. `lines` and `human` are left to the file, where the template's comment shows them. +- **Known limit: a change is scored when it passes the merge gate.** Risky settings added while a change already waits in `merge-review` with nobody on the gate do not apply to it until its next pass (a fix, a send-back or a rebuild). diff --git a/src/setup/sections/github.ts b/src/setup/sections/github.ts index 9e58359b..594fa815 100644 --- a/src/setup/sections/github.ts +++ b/src/setup/sections/github.ts @@ -1,8 +1,33 @@ +import { existsSync } from "node:fs"; +import { join } from "node:path"; import { Git } from "../../git"; import { GitHub } from "../../github"; import { errorText } from "../../shell"; import { DEFAULTS } from "../defaults"; -import type { Check, Section } from "../types"; +import type { Check, Section, SetupContext } from "../types"; + +/** Risky in any repository: CI workflows and database migrations. */ +const RISKY_ANYWHERE = [".github/**", "**/migrations/**"]; +/** Dependency and build files, suggested as risky when the repository has them at the top. */ +const RISKY_AT_TOP = ["package.json", "bun.lock", "package-lock.json", "pnpm-lock.yaml", "yarn.lock", "go.mod", "go.sum", "Cargo.toml", "Cargo.lock", "pyproject.toml", "requirements.txt", "Gemfile", "Gemfile.lock", "Dockerfile"]; + +/** + * Asks which files make a change wait for a person before it merges (gates.merge.risky.paths), when + * nobody is on the merge gate. The file's own list is the suggestion; without one, in a terminal, + * the usual risky files this repository has. --defaults never adds a list: nothing is risky until + * a person says so. + */ +async function askRisky(ctx: SetupContext): Promise<void> { + const at = ["gates", "merge", "risky", "paths"]; + const written = ctx.doc.get(at); + const own = Array.isArray(written) ? written.filter((p): p is string => typeof p === "string") : []; + const suggested = own.length ? own : ctx.ask.interactive ? [...RISKY_ANYWHERE, ...RISKY_AT_TOP.filter((f) => existsSync(join(ctx.root, f)))] : []; + ctx.ask.say("A change that touches some files (workflows, dependencies, migrations) can wait for a person before it merges, though other changes merge on their own."); + const answer = await ctx.ask.text("Files whose change needs a person before merging (comma-separated globs; - for none)", { suggestion: suggested.join(", ") || undefined, optional: true }); + const paths = answer.split(",").map((p) => p.trim()).filter(Boolean); + if (paths.length) ctx.doc.set(at, paths, { flow: true }); + else ctx.doc.clear(at); +} /** How long the remote and gh each get to answer during a check. */ export const REMOTE_CHECK_MS = 20_000; @@ -32,6 +57,7 @@ export const github: Section = { human = await ctx.ask.pick("Who approves a merge?", ["none", "status", "pr"] as const, suggestion); } ctx.doc.put(["gates", "merge", "human"], human, DEFAULTS.gates.merge.human); + if (human === "none") await askRisky(ctx); ctx.ask.say("squash: each change lands on the main branch as one commit; merge: its commits are kept, with a merge commit."); const currentMethod = ctx.doc.get(["gates", "merge", "method"]); const method = await ctx.ask.pick("Merge method", ["squash", "merge"] as const, currentMethod === "merge" || currentMethod === "squash" ? currentMethod : DEFAULTS.gates.merge.method); diff --git a/templates/config.yaml b/templates/config.yaml index 49055824..242e58c9 100644 --- a/templates/config.yaml +++ b/templates/config.yaml @@ -25,6 +25,11 @@ claude: # Gates between stages. human: status (a person edits the status line) | none; the merge gate also # takes pr (approval of its GitHub pull request). agent: true runs an independent reviewer with a fresh context. +# A risky change waits for a person before it merges, even with merge human: none. It is risky when it +# touches a file one of merge.risky's paths matches (globs: package.json only at the top, +# **/package.json anywhere), or changes more than its lines (outside intent/). Its human: pr (default; +# without a remote, the status line) or status. Nothing is risky unless set, for example: +# merge: { human: none, method: squash, risky: { paths: [".github/**", "package.json", "**/migrations/**"], lines: 1500 } } # A person always accepts an intent (draft -> accepted); by default everything after runs on its own. gates: spec: { human: none, agent: true } diff --git a/templates/intent-README.md b/templates/intent-README.md index c73c4ab3..053316c9 100644 --- a/templates/intent-README.md +++ b/templates/intent-README.md @@ -53,6 +53,8 @@ The console's buttons (approve, send back, retry, start, hold, release, priority To hold a change for a while, write a `hold:` line with a reason at the top of `intent.md`, for example `hold: waiting for legal`. The loop starts no step for it until the line is removed. The console's Hold and Release buttons write and remove it. +Some changes wait for a person before they go into the main code even when others go in on their own: those that touch files your engineers marked as needing a look (for example the files that say how the code is tested and released), or that are very large. Such a change waits at `merge-review` with a `merge_gate:` line at the top of `intent.md`, and its note names the files. With `merge_gate: pr`, approve its pull request on GitHub or set `merge-approved`; with `merge_gate: status`, read `review.md` and set `merge-approved`. The system writes and removes the line itself. Removing it by hand lets the change go in without anyone looking, so leave that to an engineer. + Sometimes the system needs a decision only a person can make. It then asks a question, and the change waits whatever its status. The `question: open` line at the top of `intent.md` is set by the loop while an agent's question waits, and `queue.md` lists the change under "Needs a person" as "Has a question". Answer it with the console's Answer button, or in the chat. To answer by hand, write your answer under the question in `conversation.md` and remove the `question:` line. `conversation.md` is the change's record of what people told the system while it worked (messages, answers, notes from sending it back) and of the questions it asked. The system writes it, and the agents that do the work read it. You can read it at any time. diff --git a/tests/unit/setup-github.test.ts b/tests/unit/setup-github.test.ts index d5b9ef54..eaeaca31 100644 --- a/tests/unit/setup-github.test.ts +++ b/tests/unit/setup-github.test.ts @@ -28,10 +28,38 @@ describe("the github section", () => { } finally { r.cleanup(); } }); + test("nobody on the merge gate: asks which files need a person, suggesting the ones this repository has", async () => { + const r = await repoWithRemote(); + try { + await Bun.write(`${r.path}/package.json`, "{}\n"); + await Bun.write(`${r.path}/go.mod`, "module x\n"); + const { text, shown } = await askSection(github, r.path, ["none", "", "squash"]); + expect(shown).toContain("Files whose change needs a person before merging (comma-separated globs; - for none) [.github/**, **/migrations/**, package.json, go.mod]"); + expect(parse(text).gates.merge).toEqual({ risky: { paths: [".github/**", "**/migrations/**", "package.json", "go.mod"] } }); + // Asked again, the file's own list is the suggestion; an answer replaces it, and - empties it. + expect(parse((await askSection(github, r.path, ["none", "", "squash"])).text).gates.merge.risky.paths).toEqual([".github/**", "**/migrations/**", "package.json", "go.mod"]); + expect(parse((await askSection(github, r.path, ["none", "infra/**, Dockerfile", "squash"])).text).gates.merge.risky.paths).toEqual(["infra/**", "Dockerfile"]); + expect((await askSection(github, r.path, ["none", "-", "squash"])).text).toBe(CONFIG); + } finally { r.cleanup(); } + }); + + test("a person on the merge gate already: the risky files are not asked about", async () => { + const r = await repoWithRemote(); + try { + const { text, shown } = await askSection(github, r.path, ["status", "squash"]); + expect(shown).not.toContain("Files whose change needs a person"); + expect(parse(text).gates.merge).toEqual({ human: "status" }); + } finally { r.cleanup(); } + }); + test("--defaults keeps merging on its own, squashed, and adds nothing", async () => { const r = await repoWithRemote(); try { expect((await askSection(github, r.path, "defaults")).text).toBe(CONFIG); + // A list already in the file is kept as it is. + const risky = `${CONFIG}gates:\n merge:\n risky:\n paths: [ .github/**, Dockerfile ]\n`; + await Bun.write(`${r.path}/loopstra/config.yaml`, risky); + expect((await askSection(github, r.path, "defaults")).text).toBe(risky); } finally { r.cleanup(); } }); From a61c303b62a5787fb851d1e93d9fd88d104c238c Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:00:26 -0500 Subject: [PATCH 12/21] feat(acceptance): the console lists acceptance.md with a change's documents Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- src/intents.ts | 2 +- tests/unit/console-detail.test.ts | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/intents.ts b/src/intents.ts index e9686c9e..043cc84c 100644 --- a/src/intents.ts +++ b/src/intents.ts @@ -155,7 +155,7 @@ export interface Intent { artifacts: Set<string>; } -export const ARTIFACTS = ["intent.md", "spec.md", "plan.md", "review.md", "outcome.md", "lessons.md", "conversation.md"] as const; +export const ARTIFACTS = ["intent.md", "spec.md", "plan.md", "acceptance.md", "review.md", "outcome.md", "lessons.md", "conversation.md"] as const; export function intentRoot(root: string): string { return join(root, "intent"); diff --git a/tests/unit/console-detail.test.ts b/tests/unit/console-detail.test.ts index fdc0bf5e..b024a560 100644 --- a/tests/unit/console-detail.test.ts +++ b/tests/unit/console-detail.test.ts @@ -51,12 +51,13 @@ describe("a change's page", () => { const trace = Trace.open(t.path); try { await Bun.write(join(t.path, "intent", "one", "spec.md"), "# Spec\n"); + await Bun.write(join(t.path, "intent", "one", "acceptance.md"), "# Acceptance tests\n"); trace.gate("one", "plan", "headings", "pass", "ok"); const fix = trace.phaseStart("one", "fix-1", "agent"); trace.phaseEnd("one", fix, { status: "fail", denied: ["Bash(git tag v1)"] }); const d = await changeDetail(t.path, trace, "one"); expect(d).toMatchObject({ slug: "one", status: "building", stage: "build", dependsOn: [] }); - expect(d!.documents).toEqual([{ name: "intent.md", url: "/docs/one/intent.md" }, { name: "spec.md", url: "/docs/one/spec.md" }]); + expect(d!.documents).toEqual([{ name: "intent.md", url: "/docs/one/intent.md" }, { name: "spec.md", url: "/docs/one/spec.md" }, { name: "acceptance.md", url: "/docs/one/acceptance.md" }]); expect(d!.phases[0]).toMatchObject({ seq: 1, name: "build", status: "success", costUsd: 0.1, denied: [], files: [{ name: "prompt.md", url: "/files/one/phases/1-build/prompt.md" }, { name: "envelope.json", url: "/files/one/phases/1-build/envelope.json" }], From 046c174ee1511ed6f03e51af6e3840041a90ea16 Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:02:28 -0500 Subject: [PATCH 13/21] feat(acceptance): the done-check reads the acceptance tests The done-check prompt gets {{acceptance}}: acceptance.md's Covered by tests list, or a sentence saying none were written. A passing acceptance test meets its criterion. A repository's older done-check.md without the variable gets the list appended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- src/intents.ts | 3 ++- src/prompts.ts | 2 +- src/stages/acceptance.ts | 8 +++++- src/stages/verify.ts | 21 +++++++++++----- templates/prompts/done-check.md | 6 ++++- tests/unit/stages-verify.test.ts | 42 ++++++++++++++++++++++++++++++++ 6 files changed, 72 insertions(+), 10 deletions(-) diff --git a/src/intents.ts b/src/intents.ts index 043cc84c..ef085081 100644 --- a/src/intents.ts +++ b/src/intents.ts @@ -119,7 +119,8 @@ export function parseIntentFile(rawText: string): IntentFile { return { frontmatter: fm.data, title, body, sections: parseSections(body) }; } -function parseSections(body: string): Record<string, string> { +/** A Markdown document's `## ` sections, by heading, each trimmed; text before the first heading is left out. */ +export function parseSections(body: string): Record<string, string> { const sections: Record<string, string> = {}; const lines = body.split(/\r?\n/); let current: string | null = null; diff --git a/src/prompts.ts b/src/prompts.ts index d2857de0..c3a4aa3b 100644 --- a/src/prompts.ts +++ b/src/prompts.ts @@ -9,7 +9,7 @@ import { templatePath } from "./package"; export const PROMPT_VARS = [ "slug", "main_branch", "intent", "priority", "spec", "plan", "review", "previous", "failure_output", "done_when", "concerns", "findings", "test_command", "test_result", "run_command", "commands", "feedback", - "conversation", + "conversation", "acceptance", ] as const; export type PromptVars = Partial<Record<(typeof PROMPT_VARS)[number], string>>; diff --git a/src/stages/acceptance.ts b/src/stages/acceptance.ts index 35ee458d..fd9f2251 100644 --- a/src/stages/acceptance.ts +++ b/src/stages/acceptance.ts @@ -2,17 +2,23 @@ import { existsSync } from "node:fs"; import { isAbsolute, join, relative } from "node:path"; import { ACCEPTANCE, writeArtifact, writeMarker, type Failure, type StepContext } from "../context"; import type { Envelope } from "../envelopes"; +import { parseSections } from "../intents"; import { passOn, type Git } from "../git"; import { agentPhase } from "../phases"; import { errorText } from "../shell"; import { bullets, KEEP_UNFINISHED, saveWork, type Artifacts } from "./shared"; /** What acceptance.md and the done-check say when the change has no acceptance tests. */ -const NONE_WRITTEN = "No acceptance tests were written for this change."; +export const NONE_WRITTEN = "No acceptance tests were written for this change."; /** The heading of acceptance.md's list of tests, which the done-check reads. */ const COVERED = "Covered by tests"; +/** acceptance.md's list of tests (see acceptanceRecord), or "" when it has none or there is no acceptance.md. */ +export function coveredByTests(acceptanceMd: string | null): string { + return acceptanceMd === null ? "" : (parseSections(acceptanceMd)[COVERED] ?? ""); +} + /** * The acceptance step (stages.build.acceptance_tests), run before the build session: a fresh session * in the change's worktree writes tests from the owner's Done when list. The runtime commits them on diff --git a/src/stages/verify.ts b/src/stages/verify.ts index 2aee5d89..31ff7c25 100644 --- a/src/stages/verify.ts +++ b/src/stages/verify.ts @@ -1,10 +1,11 @@ import { join } from "node:path"; -import { block, setStatus, writeArtifact, type Failure, type StepContext, type StepResult } from "../context"; +import { block, readArtifact, setStatus, writeArtifact, type Failure, type StepContext, type StepResult } from "../context"; import type { Envelope } from "../envelopes"; import { passOn, withDetachedWorktree } from "../git"; import { loopstraDir } from "../paths"; -import { agentPhase, alongside, mayOverlap, projectCommand } from "../phases"; +import { agentPhase, alongside, mayOverlap, phasePrompt, projectCommand, type AgentPhaseSpec } from "../phases"; import { besideWork, EARLY } from "../trace"; +import { coveredByTests, NONE_WRITTEN } from "./acceptance"; import { artifacts, bullets, humanNote, runHookCommands } from "./shared"; import { errorText } from "../shell"; @@ -12,6 +13,9 @@ const JUDGE_FAILED = "The final check could not be completed; an engineer should export const NEEDS_PERSON_DONE_NOTE = "Done. A few results need a person to confirm; see outcome.md."; const UNMET_NOTE = "The change merged, but the final check found Done when criteria that are not met. The details are in outcome.md. Decide whether to open a follow-up change, then set this one to done or closed."; +/** What the done-check is told about the acceptance tests when its prompt has no {{acceptance}} (a repository's older done-check.md). */ +const ACCEPTANCE_RULE = "These acceptance tests were written from the owner's Done when before the build. A criterion covered by an acceptance test that passes in this copy of main is met, with the test as its evidence:"; + const NO_CHECK_OUTCOME = "# Outcome\n\n## Outcome\nThe change is merged.\n\n## Evidence\nNo automatic check of the Done when criteria is configured. A person may want to confirm them.\n"; type Item = Envelope<"done-check">["evidence"][number]; @@ -85,11 +89,14 @@ function line(i: Item): string { /** * The done-check judge, run in a throwaway detached worktree of main so nothing it builds or runs - * touches the owner's checkout. Pass means no unmet items; needs-person items never block. + * touches the owner's checkout. It is given the acceptance tests written from Done when, if any. + * Pass means no unmet items; needs-person items never block. */ async function doneCheck(ctx: StepContext, a: { spec: string; review: string }): Promise<{ ok: true; items: Item[]; outcome: string } | Failure> { if (!ctx.cfg.gates.done.agent) return { ok: true, items: [], outcome: "" }; const dir = join(loopstraDir(ctx.root), "verify", ctx.slug); + // The tests written from Done when before the build (see writeAcceptance); main has them merged. + const covered = coveredByTests(await readArtifact(ctx, "acceptance.md")); let r: Awaited<ReturnType<typeof agentPhase<"done-check">>>; try { r = await withDetachedWorktree(ctx.git, dir, ctx.cfg.main_branch, async (cwd) => { @@ -99,11 +106,13 @@ async function doneCheck(ctx: StepContext, a: { spec: string; review: string }): const i = await projectCommand(ctx, install, cwd, { where: "done-check copy of main" }); if (i.code !== 0) throw new Error(`install failed: ${i.lastLine || `exit ${i.code}`}`); } - return agentPhase(ctx, { + const phase: AgentPhaseSpec & { name: "done-check" } = { name: "done-check", permissionMode: "default", tools: "read+commands", cwd, - vars: { done_when: ctx.intent.file.sections["Done when"] ?? "", spec: a.spec, review: a.review }, + vars: { done_when: ctx.intent.file.sections["Done when"] ?? "", spec: a.spec, review: a.review, acceptance: covered || NONE_WRITTEN }, skills: ctx.cfg.stages.verify.skills, - }); + }; + // The tests reach the judge even through an older done-check.md without {{acceptance}}. + return agentPhase(ctx, phase, () => phasePrompt(ctx, phase, { variable: "acceptance", note: covered, text: `${ACCEPTANCE_RULE}\n\n${covered}` })); }, (e) => ctx.trace.event(ctx.slug, "error", { where: "done-check", what: "removing the copy of main", error: errorText(e) })); } catch (e) { passOn(e); diff --git a/templates/prompts/done-check.md b/templates/prompts/done-check.md index adb9b425..836257fd 100644 --- a/templates/prompts/done-check.md +++ b/templates/prompts/done-check.md @@ -12,9 +12,13 @@ Review summary: {{review}} +Acceptance tests, written from the Done when criteria before the change was built (the build could not change them): + +{{acceptance}} + Return one item in `evidence` for each criterion, in the owner's words, with a `result`: -- `met`: you confirmed it from the repository: a test that covers it and passes, a command's output, or a file you inspected. +- `met`: you confirmed it from the repository: a test that covers it and passes, a command's output, or a file you inspected. A criterion covered by an acceptance test that passes in this copy of main is met, with the test as its evidence (say in plain words what it checks). - `unmet`: you checked, and the repository shows it is not achieved. - `needs-person`: it cannot be judged from the repository alone (for example how people feel about it, or something in production). Say briefly what a person should check. diff --git a/tests/unit/stages-verify.test.ts b/tests/unit/stages-verify.test.ts index 71663177..5d93bf73 100644 --- a/tests/unit/stages-verify.test.ts +++ b/tests/unit/stages-verify.test.ts @@ -217,6 +217,48 @@ describe("verify stage", () => { trace.close(); repo.cleanup(); }); + test("the done-check is given the acceptance tests written from Done when, and told a passing one meets its criterion", async () => { + const covered = "- add(1, 2) returns 3: `tests/accept.test.ts`, \"adds one and two\""; + const accepted = `# Acceptance tests\n\nWritten from Done when before the build. The build may not change them.\n\n## Covered by tests\n${covered}\n\n## Not covered by a test\n- People find add easy to use: needs a person.\n`; + const doneCheckPrompt = async (ctx: { runDir: string }, trace: Trace) => promptOf(ctx.runDir, trace.phases("add-numbers").find((p) => p.name === "done-check")!); + { + const { repo, ctx, trace } = await merged(); + await Bun.write(join(repo.path, "intent", "add-numbers", "acceptance.md"), accepted); + await new Git(repo.path).commitAll("acceptance"); + await ctx.reload(); + await runVerifyStep(ctx); + const prompt = await doneCheckPrompt(ctx, trace); + expect(prompt).toContain(covered); + expect(prompt).not.toContain("People find add easy to use"); + expect(prompt).toContain("A criterion covered by an acceptance test that passes in this copy of main is met"); + trace.close(); repo.cleanup(); + } + { + const { ctx, trace, repo } = await merged(); + await runVerifyStep(ctx); + expect(await doneCheckPrompt(ctx, trace)).toContain("No acceptance tests were written for this change."); + trace.close(); repo.cleanup(); + } + { + // A repository's own done-check.md without {{acceptance}} still gets the list, at its end. + const { repo, ctx, trace } = await merged({ doneCheck: "done-check" }); + await Bun.write(join(repo.path, "intent", "add-numbers", "acceptance.md"), accepted); + await new Git(repo.path).commitAll("acceptance"); + await ctx.reload(); + await runVerifyStep(ctx); + const prompt = await doneCheckPrompt(ctx, trace); + expect(prompt).toContain(`passes in this copy of main is met, with the test as its evidence:\n\n${covered}\n`); + trace.close(); repo.cleanup(); + } + { + // Without acceptance tests, an older prompt is left as it was. + const { ctx, trace, repo } = await merged({ doneCheck: "done-check" }); + await runVerifyStep(ctx); + expect(await doneCheckPrompt(ctx, trace)).not.toContain("acceptance"); + trace.close(); repo.cleanup(); + } + }); + test("the done-check runs in a throwaway copy of main, so what it builds never reaches the owner's checkout", async () => { const { repo, ctx, trace } = await merged({ doneCheck: "done-check-writes" }); await runVerifyStep(ctx); From 171e926531fe632237cf9af716a894567b641240 Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:03:10 -0500 Subject: [PATCH 14/21] feat(retro): suggest risky paths when changes merge with nobody looking Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- src/retro.ts | 8 ++++++-- tests/unit/retro.test.ts | 11 +++++++++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/src/retro.ts b/src/retro.ts index 2445aba1..1f04ba1b 100644 --- a/src/retro.ts +++ b/src/retro.ts @@ -9,6 +9,7 @@ import { SESSION_LIMIT_NOTE } from "./phases"; import { TOO_MANY_NOTE } from "./questions"; import { CHECKS_KEPT_TIMING_OUT_NOTE, TESTS_KEPT_FAILING_NOTE, VERIFY_STILL_FAILED_NOTE } from "./stages/build"; import { PR_CHECKS_FAILED_NOTE, PR_CLOSED_NOTE, REVIEWS_USED_UP, UPDATE_FAILED_NOTE } from "./stages/merge"; +import { riskySet } from "./risk"; import { REVIEW_STILL_FOUND_NOTE } from "./stages/review"; import { PR_CHECKS_NOTE } from "./stages/shared"; import { UNMET_NOTE } from "./stages/verify"; @@ -317,6 +318,7 @@ const REVIEW_BLOCKS = 2; const BUDGET_BLOCKS = 1; const LONG_WAIT_MS = DAY_MS; const QUIET_GATE_RUNS = 5; +const RISKY_MERGES = 1; const money = (usd: number) => `$${usd.toFixed(2)}`; const count = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`; @@ -360,8 +362,10 @@ export function suggestions(f: RetroFacts, cfg: Config): string[] { if (runs < QUIET_GATE_RUNS || checks.some((g) => g.passed < g.runs)) continue; out.push(`Changes waited ${duration(w.medianMs)} at the median in ${w.at}, and the ${gate} gate's automatic checks passed all ${runs} times: the person on it may not be needed (gates.${gate}.human: none).`); } - // Two more rules come with the settings they name, once those exist: - // - risky changes: gates.merge.human none, nothing set under gates.merge.risky, f.merged >= 1; + if (cfg.gates.merge.human === "none" && !riskySet(cfg) && f.merged >= RISKY_MERGES) { + out.push(`${count(f.merged, "change")} merged with nobody looking, and no file is set to need a person: set gates.merge.risky.paths (loopstra setup github suggests some) so changes to CI, dependencies or migrations wait for a person.`); + } + // One more rule comes with the setting it names, once it exists: // - acceptance tests: stages.build.acceptance_tests off, f.blocks.unmet >= 1. return out; } diff --git a/tests/unit/retro.test.ts b/tests/unit/retro.test.ts index 7b06306c..f087db9c 100644 --- a/tests/unit/retro.test.ts +++ b/tests/unit/retro.test.ts @@ -258,6 +258,16 @@ describe("suggestions", () => { expect(say()).toEqual([]); }); + test("risky changes: a merge with nobody on the gate and no risky files set suggests setting them", () => { + const merged = [mv("a", ago(2), "merge-review", "merged")]; + expect(say({ moves: merged })).toEqual([ + "1 change merged with nobody looking, and no file is set to need a person: set gates.merge.risky.paths (loopstra setup github suggests some) so changes to CI, dependencies or migrations wait for a person.", + ]); + expect(say({ moves: merged, cfg: cfgOf("gates:\n merge:\n risky:\n paths: [\".github/**\"]\n") })).toEqual([]); + expect(say({ moves: merged, cfg: cfgOf("gates:\n merge:\n risky:\n lines: 500\n") })).toEqual([]); + expect(say({ moves: merged, cfg: cfgOf("gates:\n merge:\n human: status\n") })).toEqual([]); + }); + test("waits: a person's gate with a median wait over a day, whose automatic checks all passed at least 5 times, may not need the person", () => { const merge = cfgOf("gates:\n merge:\n human: status\n"); const waits = (hours: number) => [mv("a", ago(5), "reviewing", "merge-review", MERGE_WAIT_NOTE), mv("a", ago(5, hours), "merge-review", "merge-approved")]; @@ -318,6 +328,7 @@ describe("retroText", () => { "", "Suggestions", "- 1 step hit the session spending limit of $2.00: raise claude.max_budget_usd with loopstra setup budgets.", + "- 1 change merged with nobody looking, and no file is set to need a person: set gates.merge.risky.paths (loopstra setup github suggests some) so changes to CI, dependencies or migrations wait for a person.", "", ].join("\n")); }); From f14219836f28a11e2ab89861ec6b3c17a89f40ab Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:03:38 -0500 Subject: [PATCH 15/21] feat(acceptance): document acceptance tests README (How it works, the step tables, Configuration, Steering and the test-protection hook), the owners' guide on acceptance.md, and the decisions behind them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- README.md | 18 +++++++++++------- docs/decisions.md | 15 +++++++++++++++ templates/intent-README.md | 2 ++ 3 files changed, 28 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 71d8dd6d..0c70f740 100644 --- a/README.md +++ b/README.md @@ -194,9 +194,9 @@ You can steer an agent while it works, from its tile on Agents or from its sessi - **Redirect.** The session stops what it was doing and starts again your way. It is the same session, so it keeps what it knew, and messages it had not read yet go along. - **Stop session.** After a confirm, the session ends and the change is held, so it does not start again until someone releases it. The change's controls (on its page and in the inbox) have Stop too, for everything the change is running. -Only work sessions can be messaged or redirected: intake, design, plan, build, fix, reconcile, revise and lessons. Checks (the spec check, the plan challenge, review, verify, the done-check and the cheaper judges) can only be stopped: a message would quietly change a gate. Approve or send back instead. A message that comes after its session has finished is kept for the change's next session. +Only work sessions can be messaged or redirected: intake, design, plan, acceptance, build, fix, reconcile, revise and lessons. Checks (the spec check, the plan challenge, review, verify, the done-check and the cheaper judges) can only be stopped: a message would quietly change a gate. Approve or send back instead. A message that comes after its session has finished is kept for the change's next session. -**Questions.** An agent may ask you when a decision belongs to a person and there is no sensible default. Intake, design, plan, build and revise can ask; fix and reconcile carry on the build's session, so they never ask. The step ends, and the change waits with `question: open` in `intent.md`. It costs nothing while it waits. It shows as "Has a question" in Needs attention, `status`, `queue.md` and the console's inbox, with the question, why it asks, and any options. Chat announces it too. +**Questions.** An agent may ask you when a decision belongs to a person and there is no sensible default. Intake, design, plan, build and revise can ask; fix and reconcile carry on the build's session, so they never ask, and acceptance never asks either: the Done when list is the owner's answer already. The step ends, and the change waits with `question: open` in `intent.md`. It costs nothing while it waits. It shows as "Has a question" in Needs attention, `status`, `queue.md` and the console's inbox, with the question, why it asks, and any options. Chat announces it too. Answer it with Answer, in the inbox or on the change's page. Or tell the orchestrator in chat; it asks you to confirm first, and only people who may act on changes can answer there. Or answer by hand: write your answer under the question in `conversation.md` and remove the `question:` line. On the change's next step, the agent that asked carries on with your answer. When it cannot (the change moved on meanwhile, or you answered by hand), the next session reads the answer in `conversation.md`. @@ -329,6 +329,7 @@ Each step is one session, with its own prompt (see Prompts), model (see Models f | `spec-check` | design | judges the spec | read | | `plan` | plan | writes the plan | read (plan mode) | | `plan-challenge` | plan | judges the plan | read | +| `acceptance` | build | writes tests from "Done when" before the build (with `stages.build.acceptance_tests`) | edit tests and test fixtures only, and run commands | | `build` | build | writes the code | edit, and run the project's commands | | `reconcile` | build | rewrites the plan when the build changed files it did not list | edit, and run commands | | `fix` | build | fixes failing checks, or a failed verify | edit, and run commands | @@ -385,6 +386,8 @@ flowchart LR nMergeChecks -->|pass| nMergeGate{"merge gate"} ``` +**Acceptance tests** (`stages.build.acceptance_tests`; on in repos set up with `loopstra init`, off in older ones until you set it). Before the build session, a fresh `acceptance` session in the change's worktree writes tests from the owner's "Done when" list, in the project's own test style and folders, and runs them (they should fail, since nothing is built yet). The protect-tests hook lets it edit only tests and test fixtures. The runtime commits them on the branch (`loopstra(<slug>): acceptance tests`) and records `intent/<slug>/acceptance.md` on `main`: each test under the criterion it covers, and the criteria no test can check, with why. From then on, build, fix, reconcile and revise (the merge checks' fixes too) may not edit those files: the hook refuses, and if the session then fails, the block note names the file and says an engineer can fix it on the branch. The drift check leaves them out, so they never start a reconcile. The step runs once per build: a build resumed after a stop keeps what it wrote, and a new build (from `plan-approved`, or a send-back) writes them again, keeping or updating what is on the branch. A session that fails or writes no test does not block: `acceptance.md` says no tests were written, what it wrote is dropped, and the build goes on as without the setting. The `done-check` reads the list, and counts a criterion whose acceptance test passes on `main` as met. + Every loop has a limit: `stages.build.max_fix_loops` (3) fixes while the tests fail, one fix when `verify` fails, and `stages.review.max_rounds` (2) revisions after review. Past a limit the change is blocked, and its note says so. `reconcile`, `fix` and `revise` continue the build's own session, so they know what the build did. With `speed.overlap_judges`, round 1's review runs beside `verify` on the same commit, and is used only if verify passes. **Merge checks** bring the branch up to date with `main`, make sure the tests pass (not run again when they already passed on the same code, since `main` only gained records), and read the newest review. Files changed in the change's worktree and not committed (a review `after` command, or a `lint --fix` in the build's own test runs, that rewrote them) are committed to the branch and reviewed once more first, so a formatting fix still reaches review. What the merge checks' own commands rewrite in tracked files (a `lint --fix`, a report with a timestamp) is put back, never kept, so it cannot start one review round after another; fixes the checks commit are reviewed once more. A review `after` command that changes the same files again right after they were kept blocks the change with a note naming them. New files the test, lint and build commands write (a coverage folder, a test report) are removed after each run, during the build as well; a fix session that runs the tests itself can still leave one, and the fix's commit takes it, so list such outputs in `.gitignore`. @@ -399,7 +402,7 @@ flowchart TB main -->|copied for each step| snap main -->|branched| wt snap -->|read| ro["read-only steps<br/>intake, design, spec-check,<br/>plan, plan-challenge, lessons"] - wr["writing steps<br/>build, fix, reconcile, revise"] -->|edit| wt + wr["writing steps<br/>acceptance, build, fix, reconcile, revise"] -->|edit| wt wt -->|read, run checks| chk["review, verify"] main -.->|"main's CLAUDE.md, settings,<br/>skills, MCP servers, from its commit"| wr main -.-> chk @@ -410,7 +413,7 @@ flowchart TB **Design and plan** sessions are read-only and return content; the runtime writes `spec.md`, `plan.md`, `review.md`, `outcome.md`. They (and intake, the spec and plan judges, and lessons) read a snapshot of `main`'s last commit taken once per step (`.loopstra/snapshots/<slug>`, removed when the step ends), never the main checkout: a person's uncommitted edits there are not part of a spec or plan, and `main` moving meanwhile (another change merging) does not change what one step reads. Only the build session edits code, in a worktree on its own branch `intent/<slug>`, and the runtime makes every commit and merge. The build session may run the configured commands (test, lint, build, run, install) on top of `claude.allowed_tools`. Sessions run with background tasks turned off (`CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1`), so a long command runs in the foreground and no task is left pending for the next session that resumes it. A chained command (`a && b`, `||`, `;`) is allowed whole and part by part, since Claude Code checks each part on its own. -**Claude Code's own setup comes from `main`.** Sessions that run in a change's branch (build, fix, reconcile, revise, verify, review) never load the branch's `.claude/settings.json`, `CLAUDE.md`, `.claude/rules`, `.claude/skills`, `.claude/agents`, `.claude/commands` or `.mcp.json`: a build could otherwise change what the sessions after it are told and allowed, its judges included. They load your user settings (none with `sources: repo`) and get `main`'s project setup instead, read from the step's commit of `main` into a temporary folder of the session's own (never from a checkout a session could have written): +**Claude Code's own setup comes from `main`.** Sessions that run in a change's branch (acceptance, build, fix, reconcile, revise, verify, review) never load the branch's `.claude/settings.json`, `CLAUDE.md`, `.claude/rules`, `.claude/skills`, `.claude/agents`, `.claude/commands` or `.mcp.json`: a build could otherwise change what the sessions after it are told and allowed, its judges included. They load your user settings (none with `sources: repo`) and get `main`'s project setup instead, read from the step's commit of `main` into a temporary folder of the session's own (never from a checkout a session could have written): - its `.claude/settings.json` in `--settings`, next to Loopstra's hook (both sets of hooks run); - its top-level `CLAUDE.md`, `.claude/CLAUDE.md` and `.claude/rules` through `--add-dir` (read, never edited); @@ -438,12 +441,12 @@ The full design is in `docs/superpowers/specs/2026-09-28-loopstra-design.md`, wi - `claude`: models (`cheap`, `default`, `strong`, `max`: by default the aliases `haiku`, `sonnet`, `opus` and `fable`, which Claude Code resolves to the newest model of each family; a full name such as `claude-opus-5-5` pins one), `timeout_minutes`, optional spending limits (`max_budget_usd` per session, `max_budget_usd_per_day` for the loop's sessions together; unset means no limit), and `allowed_tools` for build sessions (the configured commands are always added; anything else a build needs, like `"Bash(make *)"`, goes here). Sessions load the MCP servers, skills and CLAUDE.md files of the machine and repository they run in (in a change's branch, `main`'s: see Where sessions read and write), but without an allow rule an MCP call is refused (a session cannot ask). A server can still be connecting when a session starts, so its tools may only appear after the first turn - `agents`: which agents, subagents, teams, skills, tools and MCP tools each step's session gets (see Agents for each step) - `gates`: spec, plan, merge, done, each with `human` (`status` or `none`; merge also `pr`) and `agent` (independent reviewer); merge also `method` (`squash` or `merge`) -- `stages`: per-stage model, `steps` (a model for single steps; see Models for each step), skills, `before`/`after` commands, and loop limits +- `stages`: per-stage model, `steps` (a model for single steps; see Models for each step), skills, `before`/`after` commands, and loop limits; `build.acceptance_tests` writes tests from "Done when" before each build (see Build and review). Its default is `false`, so a repository set up before it existed is unchanged; the config `loopstra init` writes sets it to `true`. It costs one more session per build - `speed`: work that runs at the same time to save waiting; only `cheap_judges: first` changes who decides (once earned, a cheaper judge's pass stands). `overlap_judges` (on) runs the first review alongside the verifier (in a copy of the tested commit), and lessons alongside the final check; when the verifier fails, the review running beside it is stopped at once and round 1 runs after the fix. With `claude.max_budget_usd_per_day` set, they overlap only when what is left of the day covers two whole sessions (`claude.max_budget_usd`), so never without a session limit. `parallel_checks` (off) starts test, lint and build together, and stops the rest once one fails; leave it off if they write the same folder. `plan_ahead` (on) writes the first plan while the spec is being checked. The plan is stopped if the spec does not pass, and it is used only if the code on `main` it read (outside `intent/` and `loopstra/config.yaml`) is still the same commit's code when planning starts. It follows the same daily budget rule, and does not run with a person on the spec gate, without a spec judge, with plan `before` commands, or while another change is about to merge. `cheap_judges` (off) asks a cheaper model (`claude.models.default`) the same question as the spec check, the plan challenge and the review: `shadow` only records whether it agreed with the strong judge, which still decides, and `first` lets it pass something alone once it has also caught the strong judge's last 30 rejections, sending anything it rejects or is unsure of to the strong judge (`loopstra judges` shows the record). In `first`, every 5th call still runs both judges as an audit, with the strong judge deciding, so a later miss sends it back to `shadow`. Like the overlaps, `shadow` and the audits run only with room for every session at once (three when the early review runs beside the verifier, or the spec check beside a plan written ahead), so never with a daily budget and no session limit; then the strong judge decides alone, and in `first` the cheaper judge never decides alone. `think_ahead` (on) runs design and plan steps beside the changes running a step (see How it works). `max_parallel_changes` (1, up to 4) is how many changes may each run a step at the same time. Raise it only if the project's commands (test, lint, build, install) can run twice at once on this machine: no fixed ports, no shared test database, no output outside the worktree. Each command, and each session that may run commands, gets `LOOPSTRA_SLUG` and `LOOPSTRA_SLOT`: a small number (0, 1, ...) that no other change running at the same time has, so it can pick its own ports, database and Docker project names (port `3100 + LOOPSTRA_SLOT`, say). Main's health check runs alone and gets no slot. More changes at once also spend the daily budget and the subscription's usage limits faster. With `claude.max_budget_usd_per_day` set, only as many run at once as what is left of the day covers whole sessions (`claude.max_budget_usd`), so one at a time without a session limit; each overlap above also counts the sessions of the other changes running. - `signals`: how often main's health check runs - `chat`: the orchestrator's model (`model`), the model of the write-up (`steps.write-intent`), optional spending limits (`max_budget_usd_per_day`, `max_budget_usd_per_session`), and the Slack and Discord bots under `transports` (see Chat) -`loopstra setup` asks about the commands, spending limits, gates, merging, chat, models (the four names, each stage's model, and, if you say yes, a model for each of its steps) and agents (each step's agent, subagents, teams, skills, tools and MCP tools, for all steps or one step); edit the file by hand for the rest (`poll_seconds`, `allowed_tools`, stage skills and commands, speed, signals). During onboarding the `loopstra` skill lists the MCP tools its own session has, says which only read and which change things, and has you enter the ones you pick in `loopstra setup agents`. +`loopstra setup` asks about the commands, spending limits, gates (and whether to write acceptance tests, suggesting yes when the file does not say), merging, chat, models (the four names, each stage's model, and, if you say yes, a model for each of its steps) and agents (each step's agent, subagents, teams, skills, tools and MCP tools, for all steps or one step); edit the file by hand for the rest (`poll_seconds`, `allowed_tools`, stage skills and commands, speed, signals). During onboarding the `loopstra` skill lists the MCP tools its own session has, says which only read and which change things, and has you enter the ones you pick in `loopstra setup agents`. **Models for each step.** The four names are `cheap`, `default`, `strong` and `max`. Each stage has a model, and so does chat; a step can have its own under `stages.<stage>.steps.<key>` (or `chat.steps.write-intent`), set to one of the four names. A step with none of its own runs on its default below. Nothing is put on `max` by a default: only a person does that. @@ -454,6 +457,7 @@ The full design is in `docs/superpowers/specs/2026-09-28-loopstra-design.md`, wi | `spec-check` | design | `strong` | | `plan` | plan | the plan stage's model | | `plan-challenge` | plan | `strong` | +| `acceptance` | build | the build stage's model | | `build` | build | the build stage's model | | `reconcile` | build | the build stage's model | | `fix` | build | the build stage's model | @@ -531,7 +535,7 @@ loopstra customize review # copy the shipped review prompt to loopstra/prompts `customize <prompt>` copies the shipped prompt into `loopstra/prompts/` and records it in `loopstra/stamp.json`. Edit it and commit both; the loop uses it from then on. A phase reads `loopstra/prompts/<name>.md` when the repository has one, else the shipped prompt. `loopstra upgrade` merges what changes in the shipped prompt into your copy (see Upgrade). To go back to the shipped prompt, delete your copy and commit that; the next upgrade drops it from the stamp, and `customize` can make it again later. `customize` refuses a prompt the repository already has its own copy of; in a repository set up by an older Loopstra it says to run `loopstra upgrade` first, and when `loopstra/stamp.json` cannot be read (merge conflict markers, say) it says to fix and commit it first. -**The test-protection hook** stops fix sessions editing tests to make them pass. It knows a test by its folder (`tests/`, `Tests/`, `__tests__/`, `spec/`, a .NET `App.Tests/`, `App.Tests.Unit/` or `App.UnitTests/`) or its name (`a.test.ts`, `a.spec.ts`, `add_test.go`, `calc_test.py`, `test_calc.py`, `calc_test.rb`, `calc_spec.rb`, `calc_test.exs`, `calc_test.cc`, `CalcTests.cs`, `CalcTest.java`, `CalcTests.swift`). A source file whose name only mentions a test (`ab-test.ts`, `self-test.ts`, `openapi-spec.yaml`, `ABTest.java`) is not one. Tests the change itself added or changed on its branch stay editable; when a fix is refused an edit, the block note names the file. The hook comes with every session Loopstra starts for a change: it is passed to `claude` with `--settings` and runs from the installed package (`bun "<package>/templates/hooks/loopstra-protect-tests.ts"`). It does nothing outside a fix session, and nothing is written into the repository's `.claude/`. Claude Code runs it through its own shell (Git Bash on Windows), so `bun` must be on the PATH Claude Code sees. +**The test-protection hook** stops fix sessions editing tests to make them pass. It knows a test by its folder (`tests/`, `Tests/`, `__tests__/`, `spec/`, a .NET `App.Tests/`, `App.Tests.Unit/` or `App.UnitTests/`) or its name (`a.test.ts`, `a.spec.ts`, `add_test.go`, `calc_test.py`, `test_calc.py`, `calc_test.rb`, `calc_spec.rb`, `calc_test.exs`, `calc_test.cc`, `CalcTests.cs`, `CalcTest.java`, `CalcTests.swift`). A source file whose name only mentions a test (`ab-test.ts`, `self-test.ts`, `openapi-spec.yaml`, `ABTest.java`) is not one. Tests the change itself added or changed on its branch stay editable; when a fix is refused an edit, the block note names the file. The hook comes with every session Loopstra starts for a change: it is passed to `claude` with `--settings` and runs from the installed package (`bun "<package>/templates/hooks/loopstra-protect-tests.ts"`). It also keeps acceptance tests in place: the `acceptance` session may edit only tests and test fixtures (by the same folders and names), and build, fix, reconcile and revise may not edit the acceptance test files, which reach it in `LOOPSTRA_ACCEPTANCE`. It does nothing in other sessions, and nothing is written into the repository's `.claude/`. Claude Code runs it through its own shell (Git Bash on Windows), so `bun` must be on the PATH Claude Code sees. ## Upgrade diff --git a/docs/decisions.md b/docs/decisions.md index 14d28af6..a6406fec 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -888,3 +888,18 @@ A new `agents` block in `loopstra/config.yaml` gives each step an agent to play, So these sessions get main's setup, read from the step's commit of main (`git ls-tree` and `cat-file`) into a temporary folder of the session's own, removed when it ends: its settings merged into `--settings` (lists and maps joined, so both Loopstra's hook and main's run), its CLAUDE.md files and `.claude/rules` (which an added dir loads too, checked live) as a read-only `--add-dir`, its `.claude/skills`, `agents` and `commands` as the plugin `project`, and its `.mcp.json` servers less those its settings disable. A first version read them from the step's snapshot, a checkout: the review of this change found that a build can write the snapshot with any command it may run (only the edit tools are denied there), which is why agent files were already read from the commit, and it made every branch step check out all of main. The limits: main's subfolder CLAUDE.md files, and files a CLAUDE.md imports from elsewhere in the repository, do not load in these sessions; a hook or MCP server of main's that runs a script from the project folder runs the branch's copy; and main's settings can still turn hooks off or allow more, as they always could. - **A session runs on the config its prompt was rendered from.** A phase read loopstra/config.yaml three times (prompt, model, agents), so a setting saved in between could give a session agents its prompt did not name. The step now remembers the config each recent prompt was rendered from, and the session that sends the prompt uses it, as do the keys that match a prompt to earlier work (the judge comparison, the plan written ahead, the early review); a retry renders and reads again. - **Shared helpers, once.** `readText` is in `fsutil.ts`, frontmatter parsing (`splitFrontmatter`) in `agent-files.ts`, the agent and skill name rule (`ITEM_NAME`) in `agent-library.ts`, `TEAM_NAME` and `once` in `agents.ts` (which the page can import), and the empty-map cleanup is `ConfigDocument.dropEmptyMaps`. + +## Acceptance tests (2026-10-27) + +- **Off unless set; new repos get it.** `stages.build.acceptance_tests` defaults to `false` in the schema, so an upgraded repo runs exactly as before. The config `loopstra init` writes sets it to `true`, and `loopstra setup gates` asks "Write tests from Done when before each build?", suggesting the file's own value, or yes when the file does not say. So `setup --defaults` on an older config turns it on: running setup is a person choosing. It costs one more session per build. +- **A step of the build stage, with its own model and agent setup.** `acceptance` is a step key under build (its model follows the build stage's), listed before `build`, so `stages.build.steps.acceptance` and `agents.steps.acceptance` work like the others. It is a writing step, never read-only, and not a continuation of the build's session: it runs in a fresh session as its own agent, since it must not know how the build will go about it. +- **Once per build, before the build session.** It runs after the worktree is ready and the build's `before` commands ran. The run folder's `acceptance` marker (the test files, as JSON) says it ran for this build. A new build (the status was not `building`) clears it, so its step runs again and may keep or update what an earlier build's step wrote; a build resumed after a stop keeps it and does not run the step twice. What an earlier build left unsaved in the worktree is committed first (`chore: keep unfinished changes`), so the acceptance commit holds only the tests. +- **The protected files come from git, and from the tests it names.** The marker lists the files the acceptance commit added or changed (deletions left out), plus each file its answer names as a test that is in the worktree: a step that keeps an earlier build's tests unchanged writes nothing, and they must stay protected. A named path is taken relative to the worktree with forward slashes; one outside it is ignored. +- **The commit is the runtime's.** `loopstra(<slug>): acceptance tests`, with the session's `commit_message`, when it gives one, as the body. +- **Failing softly.** A session that fails (an answer of `fail`, a crash or a timeout after its retry, the spending limit) does not block the change: an `error` event says why, what it wrote is dropped from the worktree, `acceptance.md` says no tests were written, the marker is written empty so a resumed build does not try again, and the build goes on as without the setting. A stop, a question limit or an unavailable assistant still ends the step as everywhere, and the step runs again when it resumes. Only a commit that fails blocks, as the build's own commit would. +- **`acceptance.md` is for the owner.** "Covered by tests" lists each test as its criterion in the owner's words (without its last period), then its file as code and its name in quotes, as in the spec's example; "Not covered by a test" lists each criterion it could not test with why. A test whose file is not in the marker is left out. With no test, it says "No acceptance tests were written for this change." and why. It is in the console's document list. +- **Every session gets the files; the hook decides who may edit them.** `LOOPSTRA_ACCEPTANCE` is set for every session from the marker (`[]` when there is none), rather than for a chosen few, so the merge checks' fixes, which run in another stage, are covered too. The hook refuses an edit to one of them in build, fix, reconcile and revise, logs it with the other refused tests, and says "this is an acceptance test written from Done when; fix the code, not the test". In the `acceptance` phase it refuses any file that is not a test by its existing rules (test folders and names; fixtures under a test folder count). A list that cannot be read holds nothing, so a broken marker never stops a build. Paths are compared with forward slashes, and without case on Windows and macOS, as the fix rule does. +- **The block note names an acceptance test apart from other protected tests.** "It was not allowed to edit the acceptance test `tests/count.test.ts`, written from Done when before the build. If the test is wrong, an engineer can fix it on the branch intent/count-words." When a fix was refused both kinds, both sentences come before the one about the branch. +- **The drift check leaves them out.** Acceptance test files are not the plan's to list, so they never start a reconcile. They are not added to the files a change holds against other changes (see `sameFileHolds`): two changes that write the same test file are rare, and the merge brings any clash up as it does today. +- **The done-check reads the list.** `{{acceptance}}` is acceptance.md's "Covered by tests" list, or "No acceptance tests were written for this change.", and the prompt says a criterion covered by an acceptance test that passes in its copy of main is met. A repository's older `done-check.md` without the variable gets the list appended, but only when there is one, so a change without acceptance tests sees that prompt unchanged. +- **A work session that never asks.** `acceptance` can be messaged, redirected and stopped like the build, and reads `conversation.md`. It has no `question` in its answer: the Done when list is the owner's answer already. diff --git a/templates/intent-README.md b/templates/intent-README.md index c73c4ab3..ba742bad 100644 --- a/templates/intent-README.md +++ b/templates/intent-README.md @@ -63,6 +63,8 @@ A finished change also gets `lessons.md`, notes for the engineers; you can leave `spec.md` is written partly for you: its Summary, Requirements, Out of scope, Open questions, and Areas of concern say in plain words what the change will do. Its Design and Affected code sections are for the engineers. `plan.md` is written for the engineers; you do not need to read it unless the note asks you to. +Before a change is built, the system may write tests from your "Done when" list, and lists them in `acceptance.md`: each test under the item it checks, and the items no test can check (how people feel about something, say), with why. The build may not change these tests, so they hold it to what you asked, and the final check counts an item whose test passes as done. Write "Done when" items someone could check, and more of them can be tested. If `acceptance.md` says no tests were written, the change was built without them. + When a change is finished, `outcome.md` may have a section called "For a person to confirm". These are things the system could not check for itself. They never hold the change up; look at them when you can. ## Template From 3960a61c74f22d5d756357691f206c2e4302ea37 Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:07:03 -0500 Subject: [PATCH 16/21] feat(acceptance): write the template's build stage as a block map With acceptance_tests the flow map passed the YAML line width, so the stamped config no longer read back exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- templates/config.yaml | 8 +++++++- tests/unit/setup-models.test.ts | 2 +- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/templates/config.yaml b/templates/config.yaml index 13d787a9..e9daceb6 100644 --- a/templates/config.yaml +++ b/templates/config.yaml @@ -40,7 +40,13 @@ gates: stages: design: { model: strong, skills: [], before: [], after: [] } plan: { model: strong, skills: [], before: [], after: [] } - build: { model: default, skills: [], before: [], after: [], max_fix_loops: 3, acceptance_tests: true } + build: + model: default + skills: [] + before: [] + after: [] + max_fix_loops: 3 + acceptance_tests: true review: { model: strong, skills: [], before: [], after: [], max_rounds: 2 } verify: { model: default, skills: [], before: [], after: [] } diff --git a/tests/unit/setup-models.test.ts b/tests/unit/setup-models.test.ts index d4cfe17a..20442962 100644 --- a/tests/unit/setup-models.test.ts +++ b/tests/unit/setup-models.test.ts @@ -93,7 +93,7 @@ describe("the models section", () => { const before = readFileSync(configPath(repo.path), "utf8"); expect((await askSection(models, repo.path, "defaults")).text).toBe(before); const { text } = await askSection(models, repo.path, ["", "", "", "", "", "", "", "", "strong", "", "", "", "", "", "", ""]); - expect(text).toBe(before.replace("build: { model: default,", "build: { model: strong,")); + expect(text).toBe(before.replace(" build:\n model: default\n", " build:\n model: strong\n")); } finally { repo.cleanup(); } }); }); From 027a0aa3d678f6e7f31fd54d4fce85810bdaf9a6 Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:07:57 -0500 Subject: [PATCH 17/21] feat(retro): suggest acceptance tests when a change merged short of Done when Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- docs/decisions.md | 2 +- src/retro.ts | 6 ++++-- tests/unit/retro.test.ts | 8 ++++++++ 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/decisions.md b/docs/decisions.md index 3bfbe9b0..1c9fc781 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -900,7 +900,7 @@ A new `agents` block in `loopstra/config.yaml` gives each step an agent to play, - **Spending counts ended phases only.** A running phase's cost is a budget hold, not spending. "A change" is a change with a phase that ended in the window; kinds of step that cost nothing (the code phases) are not listed. Judging is every spec-check, plan-challenge and review session, the cheaper judges' and the early review's included. - **A wait at merge-review is a person's unless its note says it waits for GitHub's checks.** The other review statuses are entered only when a person decides, so every wait there is a person's. This reads what happened, whatever the config says now. A wait starts in the window and ends at the change's next move to another status; one still open counts until now and is reported as still waiting, since the longest waits are the ones that matter. A question's wait ends at an applied answer or at the change's next status move. - **Suggestion details.** Timeouts are counted from blocks, not from timeouts retried away. The longer timeout is half as long again, rounded up to 5 minutes. A person may come off a gate only when the config has one on it now, the median wait there is over 24 hours, and every automatic check of that gate in the window passed, at least 5 times in all. Whether a cheaper judge has earned going first is `judgeReport`'s new `earned` field, not its words. -- **The two rules on settings from the other parts slot in later.** "Set `gates.merge.risky`" and "turn on `stages.build.acceptance_tests`" read keys added on other branches; `suggestions` marks where they go, and `RetroFacts` already carries what they read (`merged`, `blocks.unmet`). +- **Two rules point at the other two features.** With nobody on the merge gate and nothing under `gates.merge.risky`, one merge in the window is enough to suggest risky paths: the setting costs nothing until a risky change comes. With `stages.build.acceptance_tests` off, one change blocked on an unmet Done when criterion suggests turning it on: that is the failure the tests exist to catch before the merge. - **An empty window is said in one line.** Nothing moved, no gate ran and no change's phase or question started in the window (chat alone does not count): "Nothing happened in the last 7 days.", then the suggestions, which need data and so say there are none. - **Lists of changes stop at five**, then "and N more". diff --git a/src/retro.ts b/src/retro.ts index 1f04ba1b..569fab1c 100644 --- a/src/retro.ts +++ b/src/retro.ts @@ -319,6 +319,7 @@ const BUDGET_BLOCKS = 1; const LONG_WAIT_MS = DAY_MS; const QUIET_GATE_RUNS = 5; const RISKY_MERGES = 1; +const UNMET_BLOCKS = 1; const money = (usd: number) => `$${usd.toFixed(2)}`; const count = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`; @@ -365,8 +366,9 @@ export function suggestions(f: RetroFacts, cfg: Config): string[] { if (cfg.gates.merge.human === "none" && !riskySet(cfg) && f.merged >= RISKY_MERGES) { out.push(`${count(f.merged, "change")} merged with nobody looking, and no file is set to need a person: set gates.merge.risky.paths (loopstra setup github suggests some) so changes to CI, dependencies or migrations wait for a person.`); } - // One more rule comes with the setting it names, once it exists: - // - acceptance tests: stages.build.acceptance_tests off, f.blocks.unmet >= 1. + if (!cfg.stages.build.acceptance_tests && f.blocks.unmet >= UNMET_BLOCKS) { + out.push(`${count(f.blocks.unmet, "change")} merged without meeting a Done when criterion: set stages.build.acceptance_tests to true, so tests written from Done when hold each build to it.`); + } return out; } diff --git a/tests/unit/retro.test.ts b/tests/unit/retro.test.ts index f087db9c..01b781c4 100644 --- a/tests/unit/retro.test.ts +++ b/tests/unit/retro.test.ts @@ -268,6 +268,14 @@ describe("suggestions", () => { expect(say({ moves: merged, cfg: cfgOf("gates:\n merge:\n human: status\n") })).toEqual([]); }); + test("acceptance tests: off, with a change blocked on an unmet Done when criterion, suggests turning them on", () => { + expect(say({ moves: blocks("unmet", 1) })).toEqual([ + "1 change merged without meeting a Done when criterion: set stages.build.acceptance_tests to true, so tests written from Done when hold each build to it.", + ]); + expect(say({ moves: blocks("unmet", 1), cfg: cfgOf("stages:\n build:\n acceptance_tests: true\n") })).toEqual([]); + expect(say()).toEqual([]); + }); + test("waits: a person's gate with a median wait over a day, whose automatic checks all passed at least 5 times, may not need the person", () => { const merge = cfgOf("gates:\n merge:\n human: status\n"); const waits = (hours: number) => [mv("a", ago(5), "reviewing", "merge-review", MERGE_WAIT_NOTE), mv("a", ago(5, hours), "merge-review", "merge-approved")]; From af2f1f3c7329f8ae2cc980eb6f71b601b2409f8f Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:16:25 -0500 Subject: [PATCH 18/21] docs: finish the decisions merge with main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- docs/decisions.md | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/docs/decisions.md b/docs/decisions.md index 34df43ec..16b6b73f 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -889,7 +889,27 @@ A new `agents` block in `loopstra/config.yaml` gives each step an agent to play, - **A session runs on the config its prompt was rendered from.** A phase read loopstra/config.yaml three times (prompt, model, agents), so a setting saved in between could give a session agents its prompt did not name. The step now remembers the config each recent prompt was rendered from, and the session that sends the prompt uses it, as do the keys that match a prompt to earlier work (the judge comparison, the plan written ahead, the early review); a retry renders and reads again. - **Shared helpers, once.** `readText` is in `fsutil.ts`, frontmatter parsing (`splitFrontmatter`) in `agent-files.ts`, the agent and skill name rule (`ITEM_NAME`) in `agent-library.ts`, `TEAM_NAME` and `once` in `agents.ts` (which the page can import), and the empty-map cleanup is `ConfigDocument.dropEmptyMaps`. -<<<<<<< HEAD +## Changes running at the same time without getting in each other's way (2026-10-27) + +- **Per-change temp folder, always, with a short path.** Every claimed change's commands and command-running sessions get `TMPDIR`/`TMP`/`TEMP` set to `<system temp>/loopstra-<8 hex of the repository's path>/s<slot>`, removed when the step releases the change. Not under the repository: tools make Unix sockets under `TMPDIR`, and a socket path may be at most about 104 bytes, which a deep repository path and a long slug would pass. Named by the slot, which no other running change has. A crashed step's folder is swept at the next tick's start. Claude Code itself runs with it too, which does no harm. +- **Ports handed out and checked free, 10 by default.** A change gets the lowest block (`ports_from + i × ports_per_change`) that no other claimed change holds and whose ports all bind on 127.0.0.1 at that moment. At most 50 blocks are tried; with none free the change gets no port variables and runs as before. Kept for the step, like the slot. The check is only a snapshot: something else can take a port after it, which the project's own retry or the fix loop meets as it would today. +- **Caches per slot, not per change, and opt-in.** Per-change caches would make every change fill its own install cache. A slot belongs to one running change at a time, so per-slot caches never collide and stay warm for the next change on that slot. Off by default, because sharing a cache is usually fine and isolating it costs disk. +- **`lanes.ts` still imports nothing from the rest of Loopstra.** It gained `onRelease` hooks and `isClaimed`; `src/resources.ts` registers its own clean-up there. +- **Conflicts are resolved, not blocked.** A rebase conflict used to block the change for an engineer. Now `catchUp` (src/stages/catch-up.ts) resolves it: at review start (only when main gained code, so a branch behind by records keeps the early review), in the merge checks' `up-to-date` check, and while a change waits for a person at the merge gate. +- **Squash, then replay, only on a conflict.** A clean rebase keeps the branch's commits. On a conflict the commits are squashed into one on top of main (`merge --squash` with diff3 markers), so the agent meets every conflict once against the final code, not once per commit with the same lines conflicting again; a conflict that only existed in an intermediate commit goes away. This applies with `gates.merge.method: merge` too: the branch's commits are mostly the loop's own build, fix and revise commits, and the squashed commit lists their subjects. Authors other than the runtime keep a `Co-authored-by` trailer. +- **The resolve step runs on `strong`, fresh, in the build tool set.** It does not resume the build session (that runs on the build model, and a resolution needs both sides' intent, which its prompt carries: this change's spec and plan, the conflicts with both sides, and the main commits involved with the specs of local merges named by their `<slug>: <title>` message). It must not run git that changes history or the index; the runtime stages and commits. A resolution passes only when the session says `resolved` and no conflicted file has a `<<<<<<<`, `|||||||` or `>>>>>>>` line (a bare `=======` is also a Markdown heading underline, so it is not checked). +- **Every resolution is tested, then reviewed.** At review start and while waiting: the test loop (`resolve-test-N`) runs before the review. In the merge checks the tests check runs after the resolution as usual, and the gate reports `changed`, which sends it back for a review round. The reviewer gets `{{resolution}}`: the conflicts as they were and the agent's summary. It is cleared when a review passes. +- **Resolution reviews do not use up `max_rounds`.** `roundsAllowed` is `max_rounds` plus the resolutions tried in this build; `stages.review.max_resolutions` (3) caps them, so a change cannot chase a busy main forever. +- **An interrupted resolution is undone, and does not count.** The old head is written to `catching-up` before the squash. A stop, an outage, the day's budget or a crash part-way puts the branch back (in a `finally`, and the next time the worktree is opened, or a build starts). The count goes up only once the resolve session has answered, whatever it said. +- **Failures write `conflict.md` beside `review.md`**, with the files, the conflicts with both sides, the commits involved and the reason, and leave the branch exactly as it was. +- **Waiting changes are kept mergeable only when they conflict.** A change at `merge-review` with a person on the gate (or a pull request waiting for approval) is checked with `git merge-tree --write-tree` (git 2.38+); a branch that merges cleanly is left alone even when behind, since the merge checks bring it up to date anyway. With an older git the check says nothing and resolution waits for the merge checks. Without a remote the tick makes such a change runnable (it is not otherwise); with one the watcher does it, within `max_parallel_changes` as `mergeWhenUpToDate` does. +- **An approval from before a resolution reached GitHub does not count.** GitHub keeps an approval through a push unless the repository dismisses stale ones. With `merge.human: pr`, `resolved-at` is `pending` from the resolution until the push that carries it, then that push's time; only an approval submitted after it (`latestReviews[].submittedAt` of someone with a say) merges the pull request, and `reviewDecision: APPROVED` alone is not enough then. Comparing with the resolution's own time would let an approval of the old head, given while the resolution was being reviewed, count. The wait says why (`PR_WAIT`), so the pull request does not look stuck. +- **A waiting pull request is caught up from GitHub's copy.** Before resolving, the watcher syncs main and takes commits a person added to the pull request (as `mergeWhenUpToDate` does, under the merge lock), so the resolution includes them and its push goes over them rather than blocking as "both changed". +- **Binary conflicts go to a person.** git leaves main's copy of a binary file in place with no markers, so a resolution that did not touch it would keep main's version unseen. A conflicted file with a NUL byte blocks the resolution before the session runs. +- **New files the resolve session leaves are removed** before the resolution is saved (`git clean -fd`, not `-x`), as after the test loop's runs: the change's own new files are already in the index, and the prompt says not to add files. Commits are allowed empty: a resolution may keep main's side of everything. +- **The waiting check is cached by commit.** `conflictsWhileWaiting` keeps its last answer per branch with main's and the branch's commits, so ticks and watcher looks that find neither moved run no `merge-tree`. +- **A resolution in the local merge checks holds the merge lock.** Without a remote, the merge checks run under it, so other changes' merges wait while a resolution and its tests run. Fix sessions in the merge checks already did this; moving the catch-up before the lock would leave a window where main moves again. The catch-up rebase inside `mergeIntoMain` still blocks on a conflict: under the lock, main can only have gained records there. + ## loopstra retro (2026-10-27) - **A read-only report over the trace, in plain text like `loopstra judges`.** `src/retro.ts` is pure: it builds the report from rows the trace gives (`statusChangesSince`, `gatesSince`, `phasesSince`, `eventsOfTypeSince`), the config and the judges' report, and `src/commands/retro.ts` reads them. No schema change and no new storage; it writes nothing. @@ -933,25 +953,3 @@ A new `agents` block in `loopstra/config.yaml` gives each step an agent to play, - **The drift check leaves them out.** Acceptance test files are not the plan's to list, so they never start a reconcile. They are not added to the files a change holds against other changes (see `sameFileHolds`): two changes that write the same test file are rare, and the merge brings any clash up as it does today. - **The done-check reads the list.** `{{acceptance}}` is acceptance.md's "Covered by tests" list, or "No acceptance tests were written for this change.", and the prompt says a criterion covered by an acceptance test that passes in its copy of main is met. A repository's older `done-check.md` without the variable gets the list appended, but only when there is one, so a change without acceptance tests sees that prompt unchanged. - **A work session that never asks.** `acceptance` can be messaged, redirected and stopped like the build, and reads `conversation.md`. It has no `question` in its answer: the Done when list is the owner's answer already. -======= -## Changes running at the same time without getting in each other's way (2026-10-27) - -- **Per-change temp folder, always, with a short path.** Every claimed change's commands and command-running sessions get `TMPDIR`/`TMP`/`TEMP` set to `<system temp>/loopstra-<8 hex of the repository's path>/s<slot>`, removed when the step releases the change. Not under the repository: tools make Unix sockets under `TMPDIR`, and a socket path may be at most about 104 bytes, which a deep repository path and a long slug would pass. Named by the slot, which no other running change has. A crashed step's folder is swept at the next tick's start. Claude Code itself runs with it too, which does no harm. -- **Ports handed out and checked free, 10 by default.** A change gets the lowest block (`ports_from + i × ports_per_change`) that no other claimed change holds and whose ports all bind on 127.0.0.1 at that moment. At most 50 blocks are tried; with none free the change gets no port variables and runs as before. Kept for the step, like the slot. The check is only a snapshot: something else can take a port after it, which the project's own retry or the fix loop meets as it would today. -- **Caches per slot, not per change, and opt-in.** Per-change caches would make every change fill its own install cache. A slot belongs to one running change at a time, so per-slot caches never collide and stay warm for the next change on that slot. Off by default, because sharing a cache is usually fine and isolating it costs disk. -- **`lanes.ts` still imports nothing from the rest of Loopstra.** It gained `onRelease` hooks and `isClaimed`; `src/resources.ts` registers its own clean-up there. -- **Conflicts are resolved, not blocked.** A rebase conflict used to block the change for an engineer. Now `catchUp` (src/stages/catch-up.ts) resolves it: at review start (only when main gained code, so a branch behind by records keeps the early review), in the merge checks' `up-to-date` check, and while a change waits for a person at the merge gate. -- **Squash, then replay, only on a conflict.** A clean rebase keeps the branch's commits. On a conflict the commits are squashed into one on top of main (`merge --squash` with diff3 markers), so the agent meets every conflict once against the final code, not once per commit with the same lines conflicting again; a conflict that only existed in an intermediate commit goes away. This applies with `gates.merge.method: merge` too: the branch's commits are mostly the loop's own build, fix and revise commits, and the squashed commit lists their subjects. Authors other than the runtime keep a `Co-authored-by` trailer. -- **The resolve step runs on `strong`, fresh, in the build tool set.** It does not resume the build session (that runs on the build model, and a resolution needs both sides' intent, which its prompt carries: this change's spec and plan, the conflicts with both sides, and the main commits involved with the specs of local merges named by their `<slug>: <title>` message). It must not run git that changes history or the index; the runtime stages and commits. A resolution passes only when the session says `resolved` and no conflicted file has a `<<<<<<<`, `|||||||` or `>>>>>>>` line (a bare `=======` is also a Markdown heading underline, so it is not checked). -- **Every resolution is tested, then reviewed.** At review start and while waiting: the test loop (`resolve-test-N`) runs before the review. In the merge checks the tests check runs after the resolution as usual, and the gate reports `changed`, which sends it back for a review round. The reviewer gets `{{resolution}}`: the conflicts as they were and the agent's summary. It is cleared when a review passes. -- **Resolution reviews do not use up `max_rounds`.** `roundsAllowed` is `max_rounds` plus the resolutions tried in this build; `stages.review.max_resolutions` (3) caps them, so a change cannot chase a busy main forever. -- **An interrupted resolution is undone, and does not count.** The old head is written to `catching-up` before the squash. A stop, an outage, the day's budget or a crash part-way puts the branch back (in a `finally`, and the next time the worktree is opened, or a build starts). The count goes up only once the resolve session has answered, whatever it said. -- **Failures write `conflict.md` beside `review.md`**, with the files, the conflicts with both sides, the commits involved and the reason, and leave the branch exactly as it was. -- **Waiting changes are kept mergeable only when they conflict.** A change at `merge-review` with a person on the gate (or a pull request waiting for approval) is checked with `git merge-tree --write-tree` (git 2.38+); a branch that merges cleanly is left alone even when behind, since the merge checks bring it up to date anyway. With an older git the check says nothing and resolution waits for the merge checks. Without a remote the tick makes such a change runnable (it is not otherwise); with one the watcher does it, within `max_parallel_changes` as `mergeWhenUpToDate` does. -- **An approval from before a resolution reached GitHub does not count.** GitHub keeps an approval through a push unless the repository dismisses stale ones. With `merge.human: pr`, `resolved-at` is `pending` from the resolution until the push that carries it, then that push's time; only an approval submitted after it (`latestReviews[].submittedAt` of someone with a say) merges the pull request, and `reviewDecision: APPROVED` alone is not enough then. Comparing with the resolution's own time would let an approval of the old head, given while the resolution was being reviewed, count. The wait says why (`PR_WAIT`), so the pull request does not look stuck. -- **A waiting pull request is caught up from GitHub's copy.** Before resolving, the watcher syncs main and takes commits a person added to the pull request (as `mergeWhenUpToDate` does, under the merge lock), so the resolution includes them and its push goes over them rather than blocking as "both changed". -- **Binary conflicts go to a person.** git leaves main's copy of a binary file in place with no markers, so a resolution that did not touch it would keep main's version unseen. A conflicted file with a NUL byte blocks the resolution before the session runs. -- **New files the resolve session leaves are removed** before the resolution is saved (`git clean -fd`, not `-x`), as after the test loop's runs: the change's own new files are already in the index, and the prompt says not to add files. Commits are allowed empty: a resolution may keep main's side of everything. -- **The waiting check is cached by commit.** `conflictsWhileWaiting` keeps its last answer per branch with main's and the branch's commits, so ticks and watcher looks that find neither moved run no `merge-tree`. -- **A resolution in the local merge checks holds the merge lock.** Without a remote, the merge checks run under it, so other changes' merges wait while a resolution and its tests run. Fix sessions in the merge checks already did this; moving the catch-up before the lock would leave a window where main moves again. The catch-up rebase inside `mergeIntoMain` still blocks on a conflict: under the lock, main can only have gained records there. ->>>>>>> origin/main From 15250b1fcc4c8eac21b48df8181cae3de0bc7289 Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:18:53 -0500 Subject: [PATCH 19/21] fix(acceptance): hold only tests, drop test-command output, done-check ignores a stale acceptance.md Review findings: the acceptance step protected any file it touched or named, fixture folders like testdata/ were refused, and the done-check read acceptance.md from an earlier build. The test-path rule now lives in src/test-paths.ts, shared by the hook and the step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- src/package.ts | 3 ++- src/stages/acceptance.ts | 25 ++++++++++++++++++++-- src/stages/verify.ts | 6 ++++-- src/test-paths.ts | 26 +++++++++++++++++++++++ templates/hooks/loopstra-protect-tests.ts | 13 ++++-------- tests/unit/hook.test.ts | 2 +- tests/unit/stages-build.test.ts | 21 ++++++++++++++++++ tests/unit/stages-verify.test.ts | 16 ++++++++++++++ 8 files changed, 97 insertions(+), 15 deletions(-) create mode 100644 src/test-paths.ts diff --git a/src/package.ts b/src/package.ts index d4c3feeb..0959610d 100644 --- a/src/package.ts +++ b/src/package.ts @@ -48,7 +48,8 @@ export async function stampVersion(): Promise<string> { /** * The `claude --settings` JSON that adds the test-protection hook to a session Loopstra starts. The * hook runs from the installed package by absolute path (forward slashes, for the Git Bash Claude - * Code runs hooks through on Windows); it does nothing outside a fix phase. + * Code runs hooks through on Windows). It acts in the acceptance step and in the phases that edit + * the build's code (build, fix, reconcile, revise), and does nothing in any other. */ export function hookSettings(): string { const hook = templatePath("hooks", "loopstra-protect-tests.ts").replace(/\\/g, "/"); diff --git a/src/stages/acceptance.ts b/src/stages/acceptance.ts index fd9f2251..909cf444 100644 --- a/src/stages/acceptance.ts +++ b/src/stages/acceptance.ts @@ -1,4 +1,4 @@ -import { existsSync } from "node:fs"; +import { existsSync, rmSync } from "node:fs"; import { isAbsolute, join, relative } from "node:path"; import { ACCEPTANCE, writeArtifact, writeMarker, type Failure, type StepContext } from "../context"; import type { Envelope } from "../envelopes"; @@ -6,6 +6,7 @@ import { parseSections } from "../intents"; import { passOn, type Git } from "../git"; import { agentPhase } from "../phases"; import { errorText } from "../shell"; +import { isAcceptancePath } from "../test-paths"; import { bullets, KEEP_UNFINISHED, saveWork, type Artifacts } from "./shared"; /** What acceptance.md and the done-check say when the change has no acceptance tests. */ @@ -44,6 +45,8 @@ export async function writeAcceptance(ctx: StepContext, wt: Git, a: Artifacts): writeMarker(ctx, ACCEPTANCE, "[]"); return { ok: true }; } + // The session could edit only tests (the hook); anything else changed was written by the commands it ran. + await dropNonTests(ctx, wt); const subject = `loopstra(${ctx.slug}): acceptance tests`; const said = r.envelope.commit_message.trim(); const saved = await saveWork(ctx, wt, said ? `${subject}\n\n${said}` : subject); @@ -53,7 +56,8 @@ export async function writeAcceptance(ctx: StepContext, wt: Git, a: Artifacts): const written = (await wt.run(["diff", "--name-only", "-z", "--diff-filter=d", before, "HEAD", "--"])).out.split("\0").filter(Boolean); const tests = r.envelope.tests.map((t) => ({ ...t, file: inRepo(ctx, t.file) })); const named = tests.map((t) => t.file).filter((f) => f && existsSync(join(ctx.worktreeDir, f))); - const files = [...new Set([...written, ...named])]; + // Only tests and their fixtures are held: a source file the session named (the module under test) never is. + const files = [...new Set([...written, ...named])].filter(isAcceptancePath); ctx.trace.event(ctx.slug, "command", { command: "acceptance", files, tests: tests.length, not_testable: r.envelope.not_testable.length }); await writeArtifact(ctx, "acceptance.md", acceptanceRecord({ tests, not_testable: r.envelope.not_testable }, files)); writeMarker(ctx, ACCEPTANCE, JSON.stringify(files)); @@ -80,6 +84,23 @@ async function dropUnsaved(ctx: StepContext, wt: Git): Promise<void> { } } +/** + * Puts back tracked files that are not tests, and removes new ones, before the acceptance commit: + * output of the test command the session ran (a coverage folder, a cache), never its own edits. A + * failure is traced, and the commit may then include that output. + */ +async function dropNonTests(ctx: StepContext, wt: Git): Promise<void> { + try { + const changed = (await wt.changedTrackedFiles()).filter((f) => !isAcceptancePath(f)); + if (changed.length) await wt.putBack(changed); + const added = (await wt.run(["ls-files", "--others", "--exclude-standard", "-z"])).out.split("\0").filter((f) => f && !isAcceptancePath(f)); + for (const f of added) rmSync(join(ctx.worktreeDir, f), { force: true, recursive: true }); + } catch (e) { + passOn(e); + ctx.trace.event(ctx.slug, "error", { where: "acceptance", error: errorText(e), note: "files the test command wrote were not removed; the acceptance commit may include them" }); + } +} + /** Ends a sentence with a period unless it has its own ending. */ const sentence = (s: string) => (/[.!?]$/.test(s.trim()) ? s.trim() : `${s.trim()}.`); diff --git a/src/stages/verify.ts b/src/stages/verify.ts index ee0658ea..1633f175 100644 --- a/src/stages/verify.ts +++ b/src/stages/verify.ts @@ -1,5 +1,5 @@ import { join } from "node:path"; -import { block, readArtifact, setStatus, writeArtifact, type Failure, type StepContext, type StepResult } from "../context"; +import { acceptanceFiles, block, readArtifact, setStatus, writeArtifact, type Failure, type StepContext, type StepResult } from "../context"; import type { Envelope } from "../envelopes"; import { passOn, withDetachedWorktree } from "../git"; import { loopstraDir } from "../paths"; @@ -96,7 +96,9 @@ async function doneCheck(ctx: StepContext, a: { spec: string; review: string }): if (!ctx.cfg.gates.done.agent) return { ok: true, items: [], outcome: "" }; const dir = join(loopstraDir(ctx.root), "verify", ctx.slug); // The tests written from Done when before the build (see writeAcceptance); main has them merged. - const covered = coveredByTests(await readArtifact(ctx, "acceptance.md")); + // Only while this build's marker names them: acceptance.md from an earlier build (the setting + // turned off since) lists tests this build was free to change. + const covered = acceptanceFiles(ctx).length ? coveredByTests(await readArtifact(ctx, "acceptance.md")) : ""; let r: Awaited<ReturnType<typeof agentPhase<"done-check">>>; try { r = await withDetachedWorktree(ctx.git, dir, ctx.cfg.main_branch, async (cwd) => { diff --git a/src/test-paths.ts b/src/test-paths.ts new file mode 100644 index 00000000..317c76dc --- /dev/null +++ b/src/test-paths.ts @@ -0,0 +1,26 @@ +/* + * Which paths are tests, for the test-protection hook (templates/hooks/loopstra-protect-tests.ts) and + * the acceptance step. A foundation module: it imports nothing, since the hook loads it on its own. + */ + +/** + * A test folder (tests/, Tests/, __tests__/, spec/, a .NET App.Tests/, App.Tests.Unit/ or App.UnitTests/), a test file name + * (a.test.ts, a.spec.ts; _test and _spec only for the runners that use them: add_test.go, calc_test.py, calc_spec.rb; + * test_calc.py), or a .NET/JVM class name (CalcTests.cs, CalcTest.java) or XCTest's CalcTests.swift. Class names are + * case-sensitive and need a small letter or digit before Test, so Latest.java and ABTest.java are not tests. + * `path` is relative to the project, with forward slashes. + */ +export function isTestPath(path: string): boolean { + return /(^|\/)(tests?|__tests__|spec|[^/]+\.tests?(\.\w+)?)\//i.test(path) + || /(^|\/)[^/]+\.\w*Tests?(\.\w+)?\//.test(path) + || /\.(test|spec)s?\.[a-z0-9]+$|_test\.(go|py|rb|exs|cc|cpp|c|rs)$|_spec\.rb$|(^|\/)test_[^/]+\.py$/i.test(path) + || /(^|\/)[A-Za-z0-9_]*[a-z0-9](Tests?\.(cs|fs|vb|java|kt|scala)|Tests\.swift)$/.test(path); +} + +/** + * What the acceptance step may write: a test, or a fixture its tests read from a folder no test rule + * names (Go's testdata/, fixtures/, __fixtures__/, test-utils/ or test-helpers/). + */ +export function isAcceptancePath(path: string): boolean { + return isTestPath(path) || /(^|\/)(testdata|fixtures|__fixtures__|test-(utils|helpers))\//i.test(path); +} diff --git a/templates/hooks/loopstra-protect-tests.ts b/templates/hooks/loopstra-protect-tests.ts index 2d53f6e7..9ad10485 100644 --- a/templates/hooks/loopstra-protect-tests.ts +++ b/templates/hooks/loopstra-protect-tests.ts @@ -5,6 +5,8 @@ // - fix may not edit test files, except tests the change added or changed on its branch. // Input: JSON on stdin with tool_name and tool_input. Exit 2 blocks the action and sends stderr to Claude. import { appendFileSync } from "node:fs"; +// The test rules, from the installed package beside this hook. +import { isAcceptancePath, isTestPath } from "../../src/test-paths"; const input = JSON.parse(await Bun.stdin.text()) as { tool_name?: string; tool_input?: { file_path?: string; path?: string } }; const phase = process.env.LOOPSTRA_PHASE ?? ""; // The phases that change the code, and so may not change the tests written from Done when. @@ -18,16 +20,9 @@ const caseless = process.platform === "win32" || process.platform === "darwin"; const same = (a: string) => (caseless ? a.toLowerCase() : a); let path = slashes(input.tool_input?.file_path ?? input.tool_input?.path ?? ""); if (root && same(path).startsWith(`${same(root)}/`)) path = path.slice(root.length + 1); -// A test folder (tests/, Tests/, __tests__/, spec/, a .NET App.Tests/, App.Tests.Unit/ or App.UnitTests/), a test file name -// (a.test.ts, a.spec.ts; _test and _spec only for the runners that use them: add_test.go, calc_test.py, calc_spec.rb; -// test_calc.py), or a .NET/JVM class name (CalcTests.cs, CalcTest.java) or XCTest's CalcTests.swift. Class names are -// case-sensitive and need a small letter or digit before Test, so Latest.java and ABTest.java are not tests. -const isTest = /(^|\/)(tests?|__tests__|spec|[^/]+\.tests?(\.\w+)?)\//i.test(path) - || /(^|\/)[^/]+\.\w*Tests?(\.\w+)?\//.test(path) - || /\.(test|spec)s?\.[a-z0-9]+$|_test\.(go|py|rb|exs|cc|cpp|c|rs)$|_spec\.rb$|(^|\/)test_[^/]+\.py$/i.test(path) - || /(^|\/)[A-Za-z0-9_]*[a-z0-9](Tests?\.(cs|fs|vb|java|kt|scala)|Tests\.swift)$/.test(path); +const isTest = isTestPath(path); if (phase === "acceptance") { - if (isTest) process.exit(0); + if (isAcceptancePath(path)) process.exit(0); console.error(`Loopstra: the acceptance step writes only tests and test fixtures (${path}).`); process.exit(2); } diff --git a/tests/unit/hook.test.ts b/tests/unit/hook.test.ts index 1d46229c..7e414d76 100644 --- a/tests/unit/hook.test.ts +++ b/tests/unit/hook.test.ts @@ -27,7 +27,7 @@ describe("protect-tests hook", () => { test("the acceptance step may write only tests and test fixtures", async () => { const acceptance = { LOOPSTRA_PHASE: "acceptance" }; - for (const p of ["/repo/tests/count.test.ts", "/repo/tests/fixtures/words.txt", "/repo/src/__tests__/count.ts", "C:\\repo\\tests\\count.test.ts"]) { + for (const p of ["/repo/tests/count.test.ts", "/repo/tests/fixtures/words.txt", "/repo/src/__tests__/count.ts", "C:\\repo\\tests\\count.test.ts", "/repo/pkg/calc/testdata/input.json", "/repo/fixtures/users.json", "/repo/src/__fixtures__/a.json", "/repo/test-utils/helpers.ts"]) { expect({ p, code: (await runHook({ tool_name: "Write", tool_input: { file_path: p } }, acceptance)).code }).toEqual({ p, code: 0 }); } for (const p of ["/repo/src/count.ts", "/repo/package.json", "/repo/docs/testing.md"]) { diff --git a/tests/unit/stages-build.test.ts b/tests/unit/stages-build.test.ts index 009c3427..7292ff27 100644 --- a/tests/unit/stages-build.test.ts +++ b/tests/unit/stages-build.test.ts @@ -797,6 +797,27 @@ describe("acceptance tests from Done when", () => { } finally { trace.close(); repo.cleanup(); dir.cleanup(); } }); + test("only tests are committed and held: what the test command wrote is dropped, and a source file the session names is not held", async () => { + const messy = [ + `{"type":"system","subtype":"init","session_id":"fake-acceptance","cwd":"/","tools":[],"model":"fake"}`, + `{"type":"fake_action","write":{"path":"tests/accept.test.ts","content":"import { expect, test } from \\"bun:test\\";\\ntest(\\"adds one and two\\", () => { expect(1 + 2).toBe(3); });\\n"}}`, + `{"type":"fake_action","write":{"path":"coverage/lcov.info","content":"TN:\\n"}}`, + `{"type":"result","subtype":"success","is_error":false,"session_id":"fake-acceptance","total_cost_usd":0.01,"usage":{},"structured_output":{"status":"success","summary":"wrote one test","commit_message":"","tests":[{"criterion":"add(1, 2) returns 3","file":"tests/accept.test.ts","test":"adds one and two"},{"criterion":"the coverage report","file":"coverage/lcov.info","test":"lcov"}],"not_testable":[]},"result":"ok"}`, + ].join("\n") + "\n"; + const dir = fixtureDir({ acceptance: messy }); + const { repo, ctx, trace } = await planned({ test: "bun test" }, ON); + try { + const r = await withEnv({ LOOPSTRA_FAKE_FIXTURE_DIR: dir.path }, () => runBuildStep(ctx)); + expect(r.ok).toBe(true); + expect(readMarker(ctx, "acceptance")).toBe(JSON.stringify(["tests/accept.test.ts"])); + const wt = new Git(wtOf(repo)); + const sha = (await wt.run(["log", "--format=%H", "--grep=acceptance tests", "main..HEAD"])).out.trim(); + expect((await wt.run(["show", "--name-only", "--format=", sha])).out.trim()).toBe("tests/accept.test.ts"); + const md = await Bun.file(join(repo.path, "intent", "add-numbers", "acceptance.md")).text(); + expect(md).not.toContain("coverage/lcov.info"); + } finally { trace.close(); repo.cleanup(); dir.cleanup(); } + }); + test("a fix refused an acceptance test blocks with a note that names it and says how to change it", async () => { const refused = [ `{"type":"system","subtype":"init","session_id":"fake-fix-refused","cwd":"/","tools":[],"model":"fake"}`, diff --git a/tests/unit/stages-verify.test.ts b/tests/unit/stages-verify.test.ts index 5d93bf73..af38b820 100644 --- a/tests/unit/stages-verify.test.ts +++ b/tests/unit/stages-verify.test.ts @@ -1,6 +1,7 @@ import { describe, expect, test } from "bun:test"; import { copyFileSync, existsSync } from "node:fs"; import { join } from "node:path"; +import { writeMarker } from "../../src/context"; import { Git } from "../../src/git"; import { pauseAfterUnavailable, readPause } from "../../src/heartbeat"; import { readIntent } from "../../src/intents"; @@ -226,6 +227,7 @@ describe("verify stage", () => { await Bun.write(join(repo.path, "intent", "add-numbers", "acceptance.md"), accepted); await new Git(repo.path).commitAll("acceptance"); await ctx.reload(); + writeMarker(ctx, "acceptance", JSON.stringify(["tests/accept.test.ts"])); await runVerifyStep(ctx); const prompt = await doneCheckPrompt(ctx, trace); expect(prompt).toContain(covered); @@ -239,12 +241,26 @@ describe("verify stage", () => { expect(await doneCheckPrompt(ctx, trace)).toContain("No acceptance tests were written for this change."); trace.close(); repo.cleanup(); } + { + // acceptance.md from an earlier build, when this build wrote none (the setting was turned off + // since): those tests were this build's to change, so the judge is not told they hold. + const { repo, ctx, trace } = await merged(); + await Bun.write(join(repo.path, "intent", "add-numbers", "acceptance.md"), accepted); + await new Git(repo.path).commitAll("acceptance"); + await ctx.reload(); + await runVerifyStep(ctx); + const prompt = await doneCheckPrompt(ctx, trace); + expect(prompt).toContain("No acceptance tests were written for this change."); + expect(prompt).not.toContain(covered); + trace.close(); repo.cleanup(); + } { // A repository's own done-check.md without {{acceptance}} still gets the list, at its end. const { repo, ctx, trace } = await merged({ doneCheck: "done-check" }); await Bun.write(join(repo.path, "intent", "add-numbers", "acceptance.md"), accepted); await new Git(repo.path).commitAll("acceptance"); await ctx.reload(); + writeMarker(ctx, "acceptance", JSON.stringify(["tests/accept.test.ts"])); await runVerifyStep(ctx); const prompt = await doneCheckPrompt(ctx, trace); expect(prompt).toContain(`passes in this copy of main is met, with the test as its evidence:\n\n${covered}\n`); From af3bb3c211383c05d4933d34fc25cc17a1b4040a Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:20:38 -0500 Subject: [PATCH 20/21] fix(risk, retro): review findings - Risk is scored only when nobody is on the config's merge gate, so no waiting row is written that changes nothing. - The risky note says how to stop the change, and a pull request's wait stands in for it like the other merge-gate notes (isRiskNote). - Retro leaves merge/risk out of its gate lines: waiting there is a person deciding, not a failed check. - Retro finds each move's next move in one pass instead of a scan per move. - Stale-while-waiting (from #90) reads each change's own merge gate, so a risky change waiting without a remote is kept mergeable too. - hookSettings' comment names every phase the hook acts in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- docs/decisions.md | 4 ++-- .../2026-10-27-risk-retro-acceptance-design.md | 6 +++--- src/retro.ts | 17 ++++++++++++++--- src/risk.ts | 14 +++++++++++--- src/stages/merge.ts | 5 +++-- src/stages/shared.ts | 3 ++- tests/integration/risk-gate.test.ts | 4 ++-- tests/unit/retro.test.ts | 1 + tests/unit/risk.test.ts | 8 ++++---- tests/unit/stages-review-merge.test.ts | 6 +++--- tests/unit/status.test.ts | 6 ++++++ 11 files changed, 51 insertions(+), 23 deletions(-) diff --git a/docs/decisions.md b/docs/decisions.md index 16b6b73f..4e09cae3 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -917,7 +917,7 @@ A new `agents` block in `loopstra/config.yaml` gives each step an agent to play, - **Every status move is read, the rest only the window's.** A change merged this week may have been accepted long before, so the median from accepted to merged needs its whole history. Its start is the move to accepted, or, for a change written as accepted, its first move. - **A move to the same status is not a move.** It only rewrote the note, so it neither counts as a block (or a merge) nor ends a wait. - **Blocks are told apart by the constants the runtime writes them from.** The notes that were local are exported (`TESTS_KEPT_FAILING_NOTE`, `REVIEW_STILL_FOUND_NOTE`, `UNMET_NOTE`, `SESSION_LIMIT_NOTE` and the rest), and a note's kind is the first whose constant it contains, so a reworded note cannot drift from its kind. A failed session's note (timed out, crashed, the spending limit, too many questions) is looked for first, since a stage's note may lead with its own words ("The finished change could not be checked. The assistant took too long..."). A project command that kept running past its limit after every fix counts with tests that kept failing (the fix loops ran out either way), and reviews used up after late fixes with the reviewer that kept finding problems. Any other note is listed by its first sentence, without the retry advice. -- **Gates leave out what a person did.** `done/human` and `merge/pr-approved` record a person, not a check, so they are not gate lines. A change passed a check the first time when its first row in the window passed. Lines follow the stages' order. +- **Gates leave out what a person did.** `done/human` and `merge/pr-approved` record a person, not a check, and `merge/risk` says whether a person decides (its `waiting` is no failure), so none of them is a gate line. A change passed a check the first time when its first row in the window passed. Lines follow the stages' order. - **Spending counts ended phases only.** A running phase's cost is a budget hold, not spending. "A change" is a change with a phase that ended in the window; kinds of step that cost nothing (the code phases) are not listed. Judging is every spec-check, plan-challenge and review session, the cheaper judges' and the early review's included. - **A wait at merge-review is a person's unless its note says it waits for GitHub's checks.** The other review statuses are entered only when a person decides, so every wait there is a person's. This reads what happened, whatever the config says now. A wait starts in the window and ends at the change's next move to another status; one still open counts until now and is reported as still waiting, since the longest waits are the ones that matter. A question's wait ends at an applied answer or at the change's next status move. - **Suggestion details.** Timeouts are counted from blocks, not from timeouts retried away. The longer timeout is half as long again, rounded up to 5 minutes. A person may come off a gate only when the config has one on it now, the median wait there is over 24 hours, and every automatic check of that gate in the window passed, at least 5 times in all. Whether a cheaper judge has earned going first is `judgeReport`'s new `earned` field, not its words. @@ -934,7 +934,7 @@ A new `agents` block in `loopstra/config.yaml` gives each step an agent to play, - **`merge_gate` records who decides, in the same record as the move to `merge-review`.** The pull request watcher reads the change's frontmatter fresh on every look, so a status without the field could merge an unapproved risky change. `setStatus` takes frontmatter fields to set or remove in its one record for this. The field is written only when it changes who decides, that is with `gates.merge.human: none`; a change that is not risky loses one an earlier pass wrote, in that record or in the record of its merge. It is runtime-managed, and left out when blank, like `hold`. - **One helper, `changeGates`, gives a change's own gates.** The config's, with `merge_gate` on the merge gate when the config has nobody there; it never takes a person off. `waitsForPerson`, `isRunnable` and `offers` apply it themselves, so the scheduler, `queue.md`, `status`, the console and chat need no change of their own; apply's approve and send-back check, the attention note, plan-ahead's "may merge first" and the merge stage (through `mergeHuman`) use it too. A person who deletes the line takes the change back to the config's gate: a person's edits win, as everywhere. - **Without a remote, `pr` acts as `status`.** There is no pull request to approve, so a risky change waits in `merge-review` for `merge-approved` rather than blocking. The field still says `pr`, what the config asked for, so the change waits for an approval on GitHub if a remote is added before it merges. -- **With a person on the config's merge gate, only the trace changes.** The risk row is written, the field is not, and the note is the gate's usual one. +- **With a person on the config's merge gate, risk is not scored.** It would change nothing, so no diff is read and no risk row is written (a `waiting` row that waits for nothing would read as a check that keeps failing); a stale `merge_gate` is removed and the note is the gate's usual one. The review of the branch found the first version wrote the row anyway. - **The note names the files.** Whoever approves a merge needs to know what to look at, so the owner's note names up to three of them ("and N more" after that), and says how to let the change in. A change risky only by size says how many lines changed against the limit. The trace's `merge`/`risk` row names up to ten, with the size when that tripped it: `pass` with "nothing risky", or `waiting`. No row is written when nothing is set to be risky. - **Setup asks for paths only when nobody is on the merge gate,** in the GitHub section, where the merge gate is asked. The file's own list is the suggestion; without one, `.github/**`, `**/migrations/**`, and each common dependency or build file found at the top of the repository. `-` empties the list. `--defaults` never adds one: nothing is risky until a person says so. `lines` and `human` are left to the file, where the template's comment shows them. - **Known limit: a change is scored when it passes the merge gate.** Risky settings added while a change already waits in `merge-review` with nobody on the gate do not apply to it until its next pass (a fix, a send-back or a rebuild). diff --git a/docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md b/docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md index f51a9a09..d47d2ae3 100644 --- a/docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md +++ b/docs/superpowers/specs/2026-10-27-risk-retro-acceptance-design.md @@ -59,8 +59,8 @@ The diff is the branch against main (merge base), outside `intent/`: `git diff - ### What it records -- A gate row: gate `merge`, check `risk`, result `pass` with evidence "nothing risky" (only when risky settings exist), or `waiting` with evidence naming the matched files (up to 10) and the line count when that tripped it. -- Frontmatter field `merge_gate: pr | status` on the change, written by the runtime in the same record as the status move to `merge-review`, and removed by that record when the change is not risky. Runtime-managed, left out when blank (like `hold`). It is only written when it changes who decides, that is when `gates.merge.human` is `none`. When the config already has a person on the merge gate, the gate row is still written and no field is. +- A gate row: gate `merge`, check `risk`, result `pass` with evidence "nothing risky" (only when risky settings exist and nobody is on the config's merge gate), or `waiting` with evidence naming the matched files (up to 10) and the line count when that tripped it. +- Frontmatter field `merge_gate: pr | status` on the change, written by the runtime in the same record as the status move to `merge-review`, and removed by that record when the change is not risky. Runtime-managed, left out when blank (like `hold`). It is only written when it changes who decides, that is when `gates.merge.human` is `none`. When the config already has a person on the merge gate, risk is not scored at all (changed after review: a row there would change nothing). - The note says why, in plain words: "This change touches files set to need a person before merging: .github/workflows/ci.yml and package.json. Approve the pull request on GitHub, or set status to merge-approved." (With `status`: "Read review.md. To let this change in, change the status line to merge-approved." with the same first sentence.) File names are allowed in this note (Claude's call): whoever approves a merge needs to know what to look at. At most 3 names, then "and N more". ### Who reads it @@ -95,7 +95,7 @@ Read-only. Plain text, like `loopstra judges`. Reads `.loopstra/trace.db` and th 1. **Header.** "The last 7 days (since Oct 20)." 2. **What happened.** Changes that reached main (status moved to `merged`), finished (`done`), blocked (moved to `blocked`, counted once per change), closed. The median time from `accepted` (or the first status after it) to `merged`, for changes that got there in the window. 3. **Where changes blocked.** Each block in the window, grouped by kind, most common first, with the changes named: timed out, the assistant crashed, tests kept failing, the reviewer kept finding problems, the verifier found it did not work, the spending limit, merge conflicts, checks on GitHub failed, the pull request was closed, a Done when criterion not met, a question limit, and other (with its note). The kind comes from the block note: each recognized note is matched by the constant the runtime writes it from, exported where it is local today, never by a fresh regex on wording. -4. **Gates.** One line per gate check that ran in the window (spec check, plan challenge, review findings, merge risk, pr-checks, done-check): how many times it ran, how many changes passed it the first time. From the `gates` table. +4. **Gates.** One line per gate check that ran in the window (spec check, plan challenge, review findings, pr-checks, done-check; merge risk is left out, since its `waiting` is a person deciding, not a failure): how many times it ran, how many changes passed it the first time. From the `gates` table. 5. **Spending.** Total for the loop, then by kind of step (phase names folded: `fix-2`, `test-3` → `fix`, `test`; `early-review` counts as review; `-cheap` as its judge), the five biggest, with the share of the total and the average per change. Chat on its own line. 6. **Waiting for people.** For each review status a person decides (spec-review, plan-review, merge-review, verifying) and for open questions: how many times a change waited there and the median wait. From `status_change` and `question` events. 7. **Suggestions.** Zero or more, each one sentence with its evidence and the setting to change. Rules (thresholds are named constants): diff --git a/src/retro.ts b/src/retro.ts index 569fab1c..36fca2ae 100644 --- a/src/retro.ts +++ b/src/retro.ts @@ -131,8 +131,11 @@ export function windowStart(now: Date, days: number): string { const HOUR_MS = 3_600_000; const DAY_MS = 24 * HOUR_MS; -/** Checks that record what a person did, not an automatic check. */ -const PERSON_CHECKS: ReadonlySet<string> = new Set(["done/human", "merge/pr-approved"]); +/** + * Rows that are not an automatic check that passes or fails: what a person did, and merge/risk, which + * says whether a person decides (its `waiting` is a risky change waiting for one, not a failure). + */ +const PERSON_CHECKS: ReadonlySet<string> = new Set(["done/human", "merge/pr-approved", "merge/risk"]); /** The gates in the order a change meets them. */ const GATE_ORDER = ["spec", "plan", "review", "merge", "done"]; @@ -287,10 +290,18 @@ function waits(moves: readonly StatusMove[], since: string, now: string, questio w.ms.push(Date.parse(end ?? now) - Date.parse(start)); if (end === undefined) w.open++; }; + // Each move's next move of the same change, found in one pass from the newest. + const next: (string | undefined)[] = []; + const later = new Map<string, string>(); + for (let i = moves.length - 1; i >= 0; i--) { + const m = moves[i]!; + next[i] = later.get(m.slug); + later.set(m.slug, m.ts); + } moves.forEach((m, i) => { if (m.ts < since || !(m.to in REVIEW_GATE)) return; if (m.to === "merge-review" && m.note === PR_CHECKS_NOTE) return; - add(m.to as Status, m.ts, moves.slice(i + 1).find((n) => n.slug === m.slug)?.ts); + add(m.to as Status, m.ts, next[i]); }); const answers = requests.filter((r) => { let raw: unknown; diff --git a/src/risk.ts b/src/risk.ts index 273450b3..3454f19f 100644 --- a/src/risk.ts +++ b/src/risk.ts @@ -91,6 +91,9 @@ export function riskEvidence(r: Risk): string { return parts.join("; "); } +/** The words every risk note has: what makes it one (see isRiskNote). */ +const NEEDS_PERSON = "set to need a person before merging"; + /** * The owner's note on a risky change waiting in merge-review: why (at most 3 files, so whoever * approves knows what to look at), and how to let it in. `human` is who decides: pr when there is a @@ -98,14 +101,19 @@ export function riskEvidence(r: Risk): string { */ export function riskNote(r: Risk, human: "status" | "pr"): string { const why: string[] = []; - if (r.files.length) why.push(`This change touches files set to need a person before merging: ${someOf(r.files, 3)}.`); + if (r.files.length) why.push(`This change touches files ${NEEDS_PERSON}: ${someOf(r.files, 3)}.`); if (r.overLines) { why.push(r.files.length ? `It also has ${r.lines} changed lines, more than the ${r.limit ?? 0} set.` - : `This change has ${r.lines} changed lines, more than the ${r.limit ?? 0} set to need a person before merging.`); + : `This change has ${r.lines} changed lines, more than the ${r.limit ?? 0} ${NEEDS_PERSON}.`); } const how = human === "pr" ? "Approve the pull request on GitHub, or set status to merge-approved." : "Read review.md. To let this change in, change the status line to merge-approved."; - return [...why, how].join(" "); + return [...why, how, "To stop this change, set it to closed."].join(" "); +} + +/** True when a note is one riskNote wrote (the merge gate's own words, which a pull request's wait may stand in for). */ +export function isRiskNote(note: string): boolean { + return note.includes(NEEDS_PERSON); } diff --git a/src/stages/merge.ts b/src/stages/merge.ts index e4357e3a..b7e01a23 100644 --- a/src/stages/merge.ts +++ b/src/stages/merge.ts @@ -188,10 +188,11 @@ async function passMergeGate(ctx: StepContext): Promise<StepResult> { /** * Scores the change's branch in its worktree against main (see scoreRisk) and records it as the - * merge gate's risk row; null, with no row, when the config names nothing risky. + * merge gate's risk row; null, with no row, when the config names nothing risky, or already has a + * person on the merge gate (risk would change nothing, so it is neither scored nor shown). */ async function riskOf(ctx: StepContext): Promise<Risk | null> { - if (!riskySet(ctx.cfg)) return null; + if (!riskySet(ctx.cfg) || ctx.cfg.gates.merge.human !== "none") return null; const risk = await scoreRisk(new Git(ctx.worktreeDir), ctx.cfg.main_branch, ctx.cfg); ctx.trace.gate(ctx.slug, "merge", "risk", risk.risky ? "waiting" : "pass", riskEvidence(risk)); return risk; diff --git a/src/stages/shared.ts b/src/stages/shared.ts index e2574dd6..1e13d2d9 100644 --- a/src/stages/shared.ts +++ b/src/stages/shared.ts @@ -8,6 +8,7 @@ import { changeEnv } from "../resources"; import { agentPhase, codePhase, projectCommand, type AgentPhaseResult, type AgentPhaseSpec } from "../phases"; import { WATCHING_PR, type Intent, type Status } from "../intents"; import { runDir } from "../paths"; +import { isRiskNote } from "../risk"; import { errorText } from "../shell"; /** @@ -208,7 +209,7 @@ export function prWait(root: string, intent: Intent): { wait: string; note: stri if (!WATCHING_PR.has(intent.file.frontmatter.status)) return { wait: "", note, needsPerson: false }; let wait = ""; try { wait = readFileSync(join(runDir(root, intent.slug), PR_WAIT), "utf8").trim(); } catch { /* none */ } - return { wait, note: wait && GATE_NOTES.has(note.trim()) ? "" : note, needsPerson: wait === PR_MERGED_ELSEWHERE }; + return { wait, note: wait && (GATE_NOTES.has(note.trim()) || isRiskNote(note)) ? "" : note, needsPerson: wait === PR_MERGED_ELSEWHERE }; } /** diff --git a/tests/integration/risk-gate.test.ts b/tests/integration/risk-gate.test.ts index e8f75472..499b7cdc 100644 --- a/tests/integration/risk-gate.test.ts +++ b/tests/integration/risk-gate.test.ts @@ -18,7 +18,7 @@ describe("risky changes with a GitHub remote", () => { await tick(s.repo); // review, merge checks, pull request const waiting = await intentOf(s.repo); expect(waiting).toMatchObject({ status: "merge-review", merge_gate: "pr" }); - expect(waiting.note).toBe("This change touches files set to need a person before merging: src/add.ts. Approve the pull request on GitHub, or set status to merge-approved."); + expect(waiting.note).toBe("This change touches files set to need a person before merging: src/add.ts. Approve the pull request on GitHub, or set status to merge-approved. To stop this change, set it to closed."); expect(inTrace(s.repo, (t) => t.lastGate(SLUG, "merge", "risk"))).toMatchObject({ result: "waiting", evidence: "touches src/add.ts" }); // Its checks pass, but nobody approved: it does not merge, look after look. @@ -46,7 +46,7 @@ describe("risky changes with a GitHub remote", () => { await tick(s.repo); const waiting = await intentOf(s.repo); expect(waiting).toMatchObject({ status: "merge-review", merge_gate: "status" }); - expect(waiting.note).toBe("This change touches files set to need a person before merging: src/add.ts. Read review.md. To let this change in, change the status line to merge-approved."); + expect(waiting.note).toBe("This change touches files set to need a person before merging: src/add.ts. Read review.md. To let this change in, change the status line to merge-approved. To stop this change, set it to closed."); await s.setPr({ checks: "pass", reviewDecision: "APPROVED" }); await tick(s.repo); expect((await intentOf(s.repo)).status).toBe("merge-review"); diff --git a/tests/unit/retro.test.ts b/tests/unit/retro.test.ts index 01b781c4..d7288daf 100644 --- a/tests/unit/retro.test.ts +++ b/tests/unit/retro.test.ts @@ -123,6 +123,7 @@ describe("buildRetro", () => { gateRow("b", "spec", "spec-check", "pass"), gateRow("a", "done", "human", "waiting"), gateRow("a", "merge", "pr-approved", "pass"), + gateRow("a", "merge", "risk", "waiting"), gateRow("a", "done", "done-check", "pass"), ], })); diff --git a/tests/unit/risk.test.ts b/tests/unit/risk.test.ts index 12f0b170..6ac25068 100644 --- a/tests/unit/risk.test.ts +++ b/tests/unit/risk.test.ts @@ -129,16 +129,16 @@ describe("the words", () => { test("the owner's note names up to 3 files and says who decides and how", () => { expect(riskNote(risk([".github/workflows/ci.yml", "package.json"]), "pr")).toBe( - "This change touches files set to need a person before merging: .github/workflows/ci.yml and package.json. Approve the pull request on GitHub, or set status to merge-approved.", + "This change touches files set to need a person before merging: .github/workflows/ci.yml and package.json. Approve the pull request on GitHub, or set status to merge-approved. To stop this change, set it to closed.", ); expect(riskNote(risk(["a", "b", "c", "d", "e"]), "status")).toBe( - "This change touches files set to need a person before merging: a, b, c and 2 more. Read review.md. To let this change in, change the status line to merge-approved.", + "This change touches files set to need a person before merging: a, b, c and 2 more. Read review.md. To let this change in, change the status line to merge-approved. To stop this change, set it to closed.", ); expect(riskNote(risk([], { lines: 1612, limit: 1500 }), "status")).toBe( - "This change has 1612 changed lines, more than the 1500 set to need a person before merging. Read review.md. To let this change in, change the status line to merge-approved.", + "This change has 1612 changed lines, more than the 1500 set to need a person before merging. Read review.md. To let this change in, change the status line to merge-approved. To stop this change, set it to closed.", ); expect(riskNote(risk(["a"], { lines: 1612, limit: 1500 }), "pr")).toBe( - "This change touches files set to need a person before merging: a. It also has 1612 changed lines, more than the 1500 set. Approve the pull request on GitHub, or set status to merge-approved.", + "This change touches files set to need a person before merging: a. It also has 1612 changed lines, more than the 1500 set. Approve the pull request on GitHub, or set status to merge-approved. To stop this change, set it to closed.", ); }); }); diff --git a/tests/unit/stages-review-merge.test.ts b/tests/unit/stages-review-merge.test.ts index 8ca2377e..a19d0745 100644 --- a/tests/unit/stages-review-merge.test.ts +++ b/tests/unit/stages-review-merge.test.ts @@ -397,7 +397,7 @@ describe("merge", () => { const fm = ctx.intent.file.frontmatter; // Who decides is a pull request, which without a remote is the status line. expect(fm).toMatchObject({ status: "merge-review", merge_gate: "pr" }); - expect(fm.note).toBe("This change touches files set to need a person before merging: src/add.ts. Read review.md. To let this change in, change the status line to merge-approved."); + expect(fm.note).toBe("This change touches files set to need a person before merging: src/add.ts. Read review.md. To let this change in, change the status line to merge-approved. To stop this change, set it to closed."); expect(trace.lastGate("add-numbers", "merge", "risk")).toMatchObject({ result: "waiting", evidence: "touches src/add.ts" }); expect(await onMain(repo.path, "src/add.ts")).toBe(false); // The record that moved it to merge-review wrote the field: one commit, both lines. @@ -434,14 +434,14 @@ describe("merge", () => { trace.close(); repo.cleanup(); }); - test("with a person on the config's merge gate, a risky change gets the risk row but no merge_gate", async () => { + test("with a person on the config's merge gate, risk is not scored: no risk row and no merge_gate", async () => { const { repo, ctx, trace } = await builtWith("gates:\n merge:\n human: status\n risky:\n paths: [\"src/**\"]\n human: pr\n"); await runMergeStep(ctx); const i = await readIntent(repo.path, "add-numbers"); expect(i.file.frontmatter.status).toBe("merge-review"); expect(i.file.frontmatter.merge_gate).toBeUndefined(); expect(i.file.frontmatter.note).toBe(MERGE_WAIT_NOTE); - expect(trace.lastGate("add-numbers", "merge", "risk")?.result).toBe("waiting"); + expect(trace.lastGate("add-numbers", "merge", "risk")).toBeNull(); trace.close(); repo.cleanup(); }); diff --git a/tests/unit/status.test.ts b/tests/unit/status.test.ts index 088f87af..53eeb5cd 100644 --- a/tests/unit/status.test.ts +++ b/tests/unit/status.test.ts @@ -134,6 +134,10 @@ describe("renderStatus", () => { await intent(t.path, "elsewhere", "status: merge-review\nnote: Main has my migration in progress."); await wait("elsewhere", PR_MERGED_ELSEWHERE); await intent(t.path, "open", `status: merge-review\nnote: ${PR_APPROVE_NOTE}`); + // A risky change's note is the merge gate's too: the pull request's wait stands in for it. + const risky = "This change touches files set to need a person before merging: package.json. Approve the pull request on GitHub, or set status to merge-approved. To stop this change, set it to closed."; + await intent(t.path, "risky", `status: merge-review\nmerge_gate: pr\nnote: "${risky}"`); + await wait("risky", behind); // Left by an earlier build's pull request: no longer shown once the change moved on. await intent(t.path, "rebuilt", "status: building"); await wait("rebuilt", behind); @@ -149,6 +153,8 @@ describe("renderStatus", () => { expect(block).toContain(`- Waiting for you (elsewhere): ${PR_MERGED_ELSEWHERE}`); expect(block).toContain(`- Waiting for you (open): ${PR_APPROVE_NOTE}`); expect(row("rebuilt")).not.toContain(behind); + expect(row("risky")).toContain(behind); + expect(row("risky")).not.toContain("package.json"); } finally { t.cleanup(); } From c4cf399b2ed45348e843b31e325c700c39eca22f Mon Sep 17 00:00:00 2001 From: John Flavan <johnflavan@gmail.com> Date: Tue, 27 Oct 2026 22:21:09 -0500 Subject: [PATCH 21/21] test(risk): a risky change waiting under its own gate is kept mergeable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- tests/unit/conflicts.test.ts | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/tests/unit/conflicts.test.ts b/tests/unit/conflicts.test.ts index d09d658a..6975f23d 100644 --- a/tests/unit/conflicts.test.ts +++ b/tests/unit/conflicts.test.ts @@ -270,6 +270,23 @@ describe("a change waiting for a person is kept mergeable", () => { } }, 60_000); + test("a risky change waiting under its own merge_gate, with nobody on the config's gate, is kept mergeable too", async () => { + const repo = await copyRepo(templates.get("reviewed")!.path); + const configFile = join(repo.path, "loopstra", "config.yaml"); + await Bun.write(configFile, (await Bun.file(configFile).text()).replace("human: status", "human: none")); + const intentFile = join(repo.path, "intent", "add-numbers", "intent.md"); + await Bun.write(intentFile, (await Bun.file(intentFile).text()).replace("status: merge-review", "status: merge-review\nmerge_gate: status")); + await new Git(repo.path).commitAll("risky: a person decides this one"); + const { ctx, trace } = await openRepo(repo.path); + try { + await mainMoves(repo.path); + await withResolve({ resolve: resolveFixture(RESOLVED, true) }, () => runMergeStep(ctx)); + expect(await status(repo.path)).toEqual({ status: "reviewing", note: CAUGHT_UP_NOTE }); + } finally { + trace.close(); repo.cleanup(); + } + }, 60_000); + test("main gains a change elsewhere: the waiting change is left alone", async () => { const { repo, ctx, trace } = await copy("reviewed"); try {