Skip to content

Next pass: candidate slate (A) + measurement watchlist (B) #26

Description

@Carr1005

Forward-looking slate after PR #25. Two lists with different triggers: A needs a promotion pass (guide/spec edits + a validation round); B needs a tester round or a trial, not edits. Nothing here is applied.

Supersedes the forward halves of #18 and #24 (both closed). #19 (recording/compaction) stays separate — it is tooling, not spec/guide.

A. Candidates for the next promotion pass

Ordered by strength of evidence.

  1. Harness-known identities are injected, never model-guessed. The agent-memory spec has this as an R11 sentence; the guide does not. Evidence it is needed: a clean-room regen against the current guide (regen-D) contains zero such language, so a spec generated today ships the same gap that made AC21 fail — the model, never told its thread id, guessed "current_thread" and the summarize tool honestly consolidated nothing. Second datapoint: the first wording named one tool, and an independent trial build applied it to exactly that tool, leaving summary expansion unscoped. Write it concept-anchored (identities the harness knows are injected and absent from the model-facing schema; identifiers the model legitimately holds stay model-supplied).

  2. A shipped default that no acceptance criterion exercises (D10 augmentation). Tool-description augmentation is ON by default, but the offline stub cannot return augmentation JSON and no live AC registers through a real model — so 2 of 2 builds shipped it unexercised. Compare R5, which the spec deliberately made stub-testable. Candidate rule: a Ledger default that changes behavior needs at least one AC that exercises that default, offline-testable or explicitly live.

  3. An AC depending on a fixture the spec never declares. AC7 and AC11 require a "scripted summarizer stub"; §5 declares four stubs and that is not one of them — in a section that opens "MUST create these fixtures exactly as described." 2 of 2 builds invented it, so part of the offline oracle rests on builder-authored fixture content. Candidate: §14 audit line — every fixture an AC names is declared in §5.

  4. Contract constraints with no AC behind them. SummaryRecord.description minLength 8 and R6's 8–12-word label band are stated in prose/schema and never asserted. Candidate: the existing rules↔ACs bidirectional audit extends to schema constraints that encode behavior.

  5. Prose-economy sweep. This batch grew the guide 11,041 → 12,313 words (+11.5%) and the spec 9,773 → 10,528 (+7.7%), concentrated in §0/§6.0 — the sections three testers called too dense. At this rate the guide doubles roughly every six batches. Candidate: a pass whose goal is subtraction (merge overlapping rules, move rationale out of emitted templates, delete restated mechanics) rather than new rules, with the regen properties as the safety net.

  6. F8 — fixture-corpus churn across regenerations. Parked by owner decision, carried here so it is not lost. Each regeneration authors fresh synthetic fixtures by design, so fixture identities churn and cross-generation comparisons carry noise. Not currently biting: with the Option-B pattern (surgical spec edits, no regen adoption) regenerations are evidence, not products.

B. Measurement watchlist

Needs a tester round or a disposable-clone trial. No edits pending.

  • Does the orientation block cure the density complaint? Three testers found the spec overwhelming up front (R16: "a brief, high-level overview at the top"; R18: "I needed a better explanations and instructions in the beginning"). The fix shipped in Revisions batch: on-ramp + model currency + D13 scope row (spec), six guide rules, first full live-AC run (6/6 after the thread-identity fix) #25 and is unproven with humans — a build agent reusing the prose is not evidence.
  • Does the one-question gate feel light enough? R17 objected to "the heavy question-tool ceremony", and F12 replaced 13 questions with one for the default path. Needs a human reaction, not a probe.
  • Baseline resource benchmark (wall-clock, tokens, disk) for a recommended-baseline build. This is the answer to the cost-abandonment feedback, which the owner decided to measure rather than promise in spec text.
  • Does a build actually write resolved-decisions.md? Accepted residual from Revisions batch: on-ramp + model currency + D13 scope row (spec), six guide rules, first full live-AC run (6/6 after the thread-identity fix) #25: the requirement is validated in text, the behavior is not. Rides the next trial rather than a paid probe build.
  • Ingestion idempotency — one tester hit UNIQUE errors re-running setup. The current build cannot crash that way (generated ids) but duplicates silently. Proposal recorded, unapplied: an idempotent-seed AC or an explicit "re-seeding duplicates; reset first" note.
  • arXiv 429 recurrence — transient rate limits hit a tester and our own runs; the client retries internally and R11 makes a mid-loop 429 non-fatal. Proposal recorded, unapplied: document transient-429 as expected for live (net) criteria rather than mandate backoff code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions