From eff511c53bde07bd757da15a9b8bc1b395bd19a1 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 03:54:39 -0700 Subject: [PATCH 01/27] docs: plan verified study feedback repairs --- ROADMAP.md | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tasks/S01.md | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tasks/S02.md | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tasks/S03.md | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tasks/S04.md | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 335 insertions(+) create mode 100644 ROADMAP.md create mode 100644 tasks/S01.md create mode 100644 tasks/S02.md create mode 100644 tasks/S03.md create mode 100644 tasks/S04.md diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 00000000..bf048193 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,67 @@ + + + +# Echo Study Feedback Roadmap + +## Executive summary + +This plan addresses four defects confirmed in Echo source. The audit uses commit `a93e9d82e89455ed1fa0b63447c88de544b9da26`. + +The state-root API overstates its hash boundary. The operation runner requires an unrelated Git checkout and changes its input directory. Its error hides typed Action outcomes. Recovery repeats replay prefixes and retains each basis state. + +These repairs make evidence boundaries clear, make the supplied runner usable, and remove repeated recovery work. They do not complete every proposed Echo feature. + +The user requested this checked-in plan. That request overrides the normal policy that live plans exist only in GitHub. GitHub issues and PRs remain the status authority. + +## Source and verification boundary + +Source: `FEEDBACK-echo.md`, SHA-256 `a831edf49300065e982f166dae8fd2a801f533c0e88fd8afb3e52108725ccdfa`. + +Reader source: `library/causal-computing/2026-10-07-synapse-backend-experiments/originals/FEEDBACK-echo.md`. + +The auditor read all 13 feedback sections and the affected Echo code. The original benchmark sources, WAL stores, and raw results are absent. Reported measurements remain unverified. Source inspection confirms mechanisms; it does not reproduce elapsed times. + +## Claim decisions + +| Item | Decision | Evidence and reason | +| --- | --- | --- | +| 1 | Confirmed documentation defect; S01 | `snapshot.rs:81-158` hashes reachable nodes. `echo_operation.rs:4651-4683` creates a node and attachment without an edge. `worldline_state.rs:249` incorrectly says full-state. The Merkle specification deliberately defines reachable state. Preserve that law. The patch digest binds detached writes. | +| 2 | Confirmed repeated work and retention; S04 | `trusted_runtime_host.rs:4196-4217` caches each basis state. `provenance_store.rs:1053-1077` restores a base and replays its prefix. Validation requests each distinct basis. However, `tick_history` contains snapshots, receipts, and patches, not full graph stores. The proposed snapshot explanation is incorrect. | +| 3 | Confirmed durability mechanism; no demonstrated latency defect | `submit_intent_with_runtime_wal_ack_inner` commits acceptance before return. `causal_wal.rs:5895,7860` syncs the commit file. Tick batching does not batch submission commits. The API already defines this durable ACK contract. The 20 ms figure is unverified. A batch API requires a separate accepted contract. | +| 4 | Unverified measurement; no stated size limit | WAL records retain submission, package, receipt, patch, and decision evidence. The report gives no permitted byte limit. Do not weaken retained evidence to meet an invented ratio. | +| 5 | Confirmed runner defect; S02 | `xtask/src/main.rs:454-461` discovers and enters a Git root before command parsing. The runner consumes supplied artifact paths. | +| 6 | Confirmed runner diagnostic defect; S03 | `run_edict_operation.rs:379-385` discards the outcome. `echo_operation.rs:4760-4765` retains a deterministic ResultProjectionInvalid kind. Expose that kind. New retained reason codes require a separate schema contract. | +| 7 | Confirmed declared codec boundary | `parse_input` checks shape and replacement bytes. README states that generic ingress does not validate codec-owned input schemas. The compiler cannot prove arbitrary caller JSON. Type enforcement belongs in the generated or authored adapter. No generic-core defect is established. | +| 8 | Confirmed bootstrap requirement | `build_host` reconstructs the initial lane. Replay validates the registered initial boundary. WAL docs describe bootstrap sources. A standalone open API is a feature proposal. Do not infer a broken recovery promise. | +| 9 | Broad claim rejected; narrower limitation confirmed | `TrustedRuntimeApp::observe` calls ObservationService. `ObservationAt::Tick` selects past coordinates and returns a reading envelope. `QueryBytes` optics remain unsupported. Bounded Edict setup hashes the frontier, as its current contract states. | +| 10 | Confirmed declared feature gap, already tracked | `WorldlineRuntime::fork_strand` exists. The trusted host has no durable fork crossing. WAL docs declare this limit. Existing issue #605 owns WAL-backed fork/drop work; do not create a duplicate or silently adopt that broader feature. | +| 11 | Confirmed host authority split | Cargo features distinguish internal rule authoring and trusted runtime ownership. Application handles cannot own scheduler authority. The suggested convenience crate is a product proposal. Its unavailable harness does not establish the claimed import count. | +| 12 | Pins confirmed; defect rejected | `hello-echo/producers.lock.json` pins exact producers. Its README requires those revisions and explains how pins advance. A refusal on another main revision enforces reproducibility. No automatic latest-main compatibility promise exists. | +| 13 | Source-reported praise | Existing runner tests cover these mechanisms. The reported 22,100 Actions and timings remain unverified without raw results. | + +Code paths are relative to this repository, except the explicitly named hello-echo consumer. All conclusions describe the audited revision. + +## Execution sequence + +- [ ] [S01: Describe the reachable-state boundary of WorldlineState::state_root](tasks/S01.md) — [issue #754](https://github.com/flyingrobots/echo/issues/754) +- [ ] [S02: Run xtask run-edict-operation without a Git checkout or directory change](tasks/S02.md) — [issue #755](https://github.com/flyingrobots/echo/issues/755) +- [ ] [S03: Report the typed Action obstruction when the operation runner cannot commit](tasks/S03.md) — [issue #756](https://github.com/flyingrobots/echo/issues/756) +- [ ] [S04: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery](tasks/S04.md) — [issue #757](https://github.com/flyingrobots/echo/issues/757) + +Execute S01, S02, S03, then S04. Each issue produces one independently mergeable PR. This order puts smaller contract and runner repairs before recovery work. + +The dependency graph has four vertices and zero required edges. All four tasks form one antichain at the audited revision. The chosen sequence serializes the shared worker; it does not imply a code prerequisite. + +Each task owns its stated defect. S01 owns hash-boundary text. S02 owns command directory behavior. S03 owns runner outcome errors. S04 owns recovery work and retention. No task owns a second task's requirements. + +## Verification and merge rules + +Use the existing `echo-read-runtime` worker and stable `/lease-target` cache. Use `/Users/Shared/git-locks/workstation.git` with `host/heavy-work` and `host/docker/echo-read-runtime/`. The monitor also inspects `echo-provider-builder`; reserve its key during validation. + +Keep the 20 GiB build, 4 GiB data, and 128 MiB log limits. Keep the 50 GiB host and VM free-space floors. Use copied source, four CPUs, 6 GiB RAM, bounded logs, process deadlines, and the fail-closed guard. Do not use host test fallbacks. + +Record before/after evidence for S01. Record executable RED/GREEN for behavior repairs. Use deterministic replay-work counters for S04. Do not claim a timing result from source inspection. + +Each PR must pass its relevant tests, linters, hosted required checks, and current-head Code Lawyer and agy reviews. Preserve issue, PR, and merge-commit linkage. Do not amend, rebase, or force-push. The user has authorized ordinary pushes and merges. + +STE prose follows the installed ASD-STE100 guide. Full dictionary conformance is unverified. diff --git a/tasks/S01.md b/tasks/S01.md new file mode 100644 index 00000000..278e4ec8 --- /dev/null +++ b/tasks/S01.md @@ -0,0 +1,67 @@ +--- +id: S01 +type: Feature +title: Describe the reachable-state boundary of WorldlineState::state_root +issue: https://github.com/flyingrobots/echo/issues/754 +evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +depends_on: [] +--- + +# Feature + +```text +Execute https://github.com/flyingrobots/echo/issues/754. Verify the cited source. Run the named witness in the shared guarded Docker worker. Record RED evidence. Apply the smallest repair. Update current documents. Run relevant checks. Commit and push the repair. Open one PR. Obtain current-head Code Lawyer and agy approval. Merge only after all required gates pass. Preserve causal authority and unrelated work. Stop when the acceptance criteria pass. +``` + +## 1. Background Context + +Feedback item 1 in `FEEDBACK-echo.md` describes this problem. Echo source at `a93e9d82e89455ed1fa0b63447c88de544b9da26` confirms the mechanism. + +Source: https://github.com/flyingrobots/echo/blob/a93e9d82e89455ed1fa0b63447c88de544b9da26/crates/warp-core/src/worldline_state.rs#L249 + +The original experiment files are absent. The reported timings and counts are source claims, not reproduced measurements. + +## 2. Problem Description + +The API calls state_root() a full-state hash. The implementation excludes nodes that the root cannot reach. Create-if-absent produces such nodes. + +## 3. Proposed Solution + +Correct the API contract and the operation documentation. Preserve the version-1 hash law. Explain that patch and commit identities retain detached writes. + +## 4. Prerequisites + +- [ ] Use the shared guarded Docker worker for executable checks. +- [ ] Record RED evidence before the code repair, or before/after evidence for a documentation repair. + +No other PR must merge first. Execution order does not establish a dependency. + +## 5. Scope + +In: This confirmed defect, its smallest witness, current owning documentation, and CHANGELOG when behavior changes. + +Out: Application-specific APIs, provider schema changes, new hash versions, and unrelated review findings. + +## 6. Acceptance Criteria + +The API states the reachable-state boundary. The operation guide states that a new detached cell can leave the state root unchanged. A witness binds the write through its patch and commit. + +## 7. Definition of Done + +The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact PR head. The PR merges with issue linkage. + +## 8. Test Plan + +Golden: `Inspect the old and new API text. Run a focused warp-core test that creates two detached cells, compares roots, and checks distinct retained patches and commits.` + +Edges: Reachable nodes, detached nodes, descended instances, and duplicate no-mutation evidence. + +Known failure modes: The API calls state_root() a full-state hash. The implementation excludes nodes that the root cannot reach. Create-if-absent produces such nodes. + +Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. + +## 9. Stakeholders + +James Ross owns Echo. Echo host authors need correct evidence and clear errors. Application authors need the documented runner contract. + +Related: #699. The existing target-value digest remains necessary. diff --git a/tasks/S02.md b/tasks/S02.md new file mode 100644 index 00000000..bacd8f9a --- /dev/null +++ b/tasks/S02.md @@ -0,0 +1,67 @@ +--- +id: S02 +type: Feature +title: Run xtask run-edict-operation without a Git checkout or directory change +issue: https://github.com/flyingrobots/echo/issues/755 +evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +depends_on: [] +--- + +# Feature + +```text +Execute https://github.com/flyingrobots/echo/issues/755. Verify the cited source. Run the named witness in the shared guarded Docker worker. Record RED evidence. Apply the smallest repair. Update current documents. Run relevant checks. Commit and push the repair. Open one PR. Obtain current-head Code Lawyer and agy approval. Merge only after all required gates pass. Preserve causal authority and unrelated work. Stop when the acceptance criteria pass. +``` + +## 1. Background Context + +Feedback item 5 in `FEEDBACK-echo.md` describes this problem. Echo source at `a93e9d82e89455ed1fa0b63447c88de544b9da26` confirms the mechanism. + +Source: https://github.com/flyingrobots/echo/blob/a93e9d82e89455ed1fa0b63447c88de544b9da26/xtask/src/main.rs#L454 + +The original experiment files are absent. The reported timings and counts are source claims, not reproduced measurements. + +## 2. Problem Description + +main() resolves a Git root and changes the current directory before it parses the command. The operation runner needs supplied artifact paths. It fails outside Git and resolves relative paths from a different directory inside Git. + +## 3. Proposed Solution + +Parse the command first. Keep the caller directory for run-edict-operation. Resolve the Git root only for commands that require repository paths. + +## 4. Prerequisites + +- [ ] Use the shared guarded Docker worker for executable checks. +- [ ] Record RED evidence before the code repair, or before/after evidence for a documentation repair. + +No other PR must merge first. Execution order does not establish a dependency. + +## 5. Scope + +In: This confirmed defect, its smallest witness, current owning documentation, and CHANGELOG when behavior changes. + +Out: Application-specific APIs, provider schema changes, new hash versions, and unrelated review findings. + +## 6. Acceptance Criteria + +The runner succeeds outside Git with absolute paths. It resolves relative artifact paths from the caller directory. Existing repository commands retain their root policy. + +## 7. Definition of Done + +The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact PR head. The PR merges with issue linkage. + +## 8. Test Plan + +Golden: `cargo test -p xtask --test run_edict_operation. Add cases for a directory outside Git and a nested directory inside an unrelated repository.` + +Edges: Absolute paths, relative paths, nested directories, malformed inputs, and repository commands. + +Known failure modes: main() resolves a Git root and changes the current directory before it parses the command. The operation runner needs supplied artifact paths. It fails outside Git and resolves relative paths from a different directory inside Git. + +Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. + +## 9. Stakeholders + +James Ross owns Echo. Echo host authors need correct evidence and clear errors. Application authors need the documented runner contract. + +Related: #531, which describes repository-root discovery for repository commands. diff --git a/tasks/S03.md b/tasks/S03.md new file mode 100644 index 00000000..cc438a08 --- /dev/null +++ b/tasks/S03.md @@ -0,0 +1,67 @@ +--- +id: S03 +type: Feature +title: Report the typed Action obstruction when the operation runner cannot commit +issue: https://github.com/flyingrobots/echo/issues/756 +evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +depends_on: [] +--- + +# Feature + +```text +Execute https://github.com/flyingrobots/echo/issues/756. Verify the cited source. Run the named witness in the shared guarded Docker worker. Record RED evidence. Apply the smallest repair. Update current documents. Run relevant checks. Commit and push the repair. Open one PR. Obtain current-head Code Lawyer and agy approval. Merge only after all required gates pass. Preserve causal authority and unrelated work. Stop when the acceptance criteria pass. +``` + +## 1. Background Context + +Feedback item 6 in `FEEDBACK-echo.md` describes this problem. Echo source at `a93e9d82e89455ed1fa0b63447c88de544b9da26` confirms the mechanism. + +Source: https://github.com/flyingrobots/echo/blob/a93e9d82e89455ed1fa0b63447c88de544b9da26/xtask/src/run_edict_operation.rs#L379 + +The original experiment files are absent. The reported timings and counts are source claims, not reproduced measurements. + +## 2. Problem Description + +The runner discards a non-committed Action outcome. It reports only that the scheduler did not publish a committed outcome. ResultProjectionInvalid therefore disappears from the caller error. + +## 3. Proposed Solution + +Include the typed outcome in the runner error. Distinguish an absent outcome from an obstruction. Keep retained obstruction encoding unchanged. + +## 4. Prerequisites + +- [ ] Use the shared guarded Docker worker for executable checks. +- [ ] Record RED evidence before the code repair, or before/after evidence for a documentation repair. + +No other PR must merge first. Execution order does not establish a dependency. + +## 5. Scope + +In: This confirmed defect, its smallest witness, current owning documentation, and CHANGELOG when behavior changes. + +Out: Application-specific APIs, provider schema changes, new hash versions, and unrelated review findings. + +## 6. Acceptance Criteria + +An output-bound refusal names ResultProjectionInvalid. The runner emits no successful report. Existing durable obstruction and duplicate checks still pass. + +## 7. Definition of Done + +The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact PR head. The PR merges with issue linkage. + +## 8. Test Plan + +Golden: `cargo test -p xtask --test run_edict_operation. Use a canonical input whose retained projection exceeds its declared output bound.` + +Edges: Missing outcome, projection refusal, committed outcome, and duplicate obstruction. + +Known failure modes: The runner discards a non-committed Action outcome. It reports only that the scheduler did not publish a committed outcome. ResultProjectionInvalid therefore disappears from the caller error. + +Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. + +## 9. Stakeholders + +James Ross owns Echo. Echo host authors need correct evidence and clear errors. Application authors need the documented runner contract. + +Related: #696. Stable projection reason sub-codes are a separate schema proposal, not required for this repair. diff --git a/tasks/S04.md b/tasks/S04.md new file mode 100644 index 00000000..46aee711 --- /dev/null +++ b/tasks/S04.md @@ -0,0 +1,67 @@ +--- +id: S04 +type: Feature +title: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery +issue: https://github.com/flyingrobots/echo/issues/757 +evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +depends_on: [] +--- + +# Feature + +```text +Execute https://github.com/flyingrobots/echo/issues/757. Verify the cited source. Run the named witness in the shared guarded Docker worker. Record RED evidence. Apply the smallest repair. Update current documents. Run relevant checks. Commit and push the repair. Open one PR. Obtain current-head Code Lawyer and agy approval. Merge only after all required gates pass. Preserve causal authority and unrelated work. Stop when the acceptance criteria pass. +``` + +## 1. Background Context + +Feedback item 2 in `FEEDBACK-echo.md` describes this problem. Echo source at `a93e9d82e89455ed1fa0b63447c88de544b9da26` confirms the mechanism. + +Source: https://github.com/flyingrobots/echo/blob/a93e9d82e89455ed1fa0b63447c88de544b9da26/crates/warp-core/src/trusted_runtime_host.rs#L4196 + +The original experiment files are absent. The reported timings and counts are source claims, not reproduced measurements. + +## 2. Problem Description + +Recovery caches a full WorldlineState for every distinct basis tick. Each cache miss replays from the initial boundary when no checkpoint exists. Consecutive bases repeat all earlier patches and retain growing graph and tick-history copies. + +## 3. Proposed Solution + +Process required coordinates in causal order with bounded replay state. Reuse verified prefixes. Preserve all basis, patch, receipt, decision, conflict, and initial-boundary checks. + +## 4. Prerequisites + +- [ ] Use the shared guarded Docker worker for executable checks. +- [ ] Record RED evidence before the code repair, or before/after evidence for a documentation repair. + +No other PR must merge first. Execution order does not establish a dependency. + +## 5. Scope + +In: This confirmed defect, its smallest witness, current owning documentation, and CHANGELOG when behavior changes. + +Out: Application-specific APIs, provider schema changes, new hash versions, and unrelated review findings. + +## 6. Acceptance Criteria + +A multi-tick recovery witness counts applied replay patches and retained replay states. Counts grow with required history, not the sum of every prefix. Poisoned bases and reordered or incomplete outcomes still refuse. + +## 7. Definition of Done + +The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact PR head. The PR merges with issue linkage. + +## 8. Test Plan + +Golden: `cargo test -p warp-core --features native_rule_bootstrap,trusted_runtime,host_test --test executable_operation_pipeline_tests. Add deterministic work counters for consecutive ticks and mixed basis coordinates.` + +Edges: Shared bases, stale bases, multiple worldlines, checkpoints, batches, conflicts, and corrupted retained evidence. + +Known failure modes: Recovery caches a full WorldlineState for every distinct basis tick. Each cache miss replays from the initial boundary when no checkpoint exists. Consecutive bases repeat all earlier patches and retain growing graph and tick-history copies. + +Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. + +## 9. Stakeholders + +James Ross owns Echo. Echo host authors need correct evidence and clear errors. Application authors need the documented runner contract. + +Related: #751 concerns inverse-intent recovery and has separate scope. No dependency is established. From 59886b1bd831f69320c0726dd64778568c34cf73 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 04:07:36 -0700 Subject: [PATCH 02/27] docs: add experimental Keep tasks and preserve task metadata --- CHANGELOG.md | 2 + ROADMAP.md | 33 +++++++++++- docs/DOCUMENTATION_STANDARDS.md | 2 + scripts/ensure_spdx.sh | 38 ++++++++++++-- scripts/tests/spdx_frontmatter_test.sh | 57 +++++++++++++++++++++ tasks/K01.md | 70 +++++++++++++++++++++++++ tasks/K02.md | 70 +++++++++++++++++++++++++ tasks/K03.md | 71 ++++++++++++++++++++++++++ tasks/K04.md | 69 +++++++++++++++++++++++++ tasks/K05.md | 69 +++++++++++++++++++++++++ tasks/K06.md | 69 +++++++++++++++++++++++++ tasks/S01.md | 4 +- tasks/S02.md | 4 +- tasks/S03.md | 4 +- tasks/S04.md | 6 ++- 15 files changed, 556 insertions(+), 12 deletions(-) create mode 100755 scripts/tests/spdx_frontmatter_test.sh create mode 100644 tasks/K01.md create mode 100644 tasks/K02.md create mode 100644 tasks/K03.md create mode 100644 tasks/K04.md create mode 100644 tasks/K05.md create mode 100644 tasks/K06.md diff --git a/CHANGELOG.md b/CHANGELOG.md index d99c3899..71c4af93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ ### Fixed +- The SPDX checker preserves Markdown frontmatter when it checks or repairs license headers. + - `ReadView::at` validates the aperture and derives the selected current frontier basis with one state hash. `ReadView::new` still checks an independently selected expected basis. Both retain the same aperture and trust boundaries. diff --git a/ROADMAP.md b/ROADMAP.md index bf048193..c5072128 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -9,7 +9,7 @@ This plan addresses four defects confirmed in Echo source. The audit uses commit The state-root API overstates its hash boundary. The operation runner requires an unrelated Git checkout and changes its input directory. Its error hides typed Action outcomes. Recovery repeats replay prefixes and retains each basis state. -These repairs make evidence boundaries clear, make the supplied runner usable, and remove repeated recovery work. They do not complete every proposed Echo feature. +These repairs make evidence boundaries clear, make the supplied runner usable, and remove repeated recovery work. They do not complete every proposed Echo feature. The separate Keep sequence below supplies the requested CAS integration plan. The user requested this checked-in plan. That request overrides the normal policy that live plans exist only in GitHub. GitHub issues and PRs remain the status authority. @@ -50,10 +50,39 @@ Code paths are relative to this repository, except the explicitly named hello-ec Execute S01, S02, S03, then S04. Each issue produces one independently mergeable PR. This order puts smaller contract and runner repairs before recovery work. -The dependency graph has four vertices and zero required edges. All four tasks form one antichain at the audited revision. The chosen sequence serializes the shared worker; it does not imply a code prerequisite. +The feedback-repair graph has four vertices and zero required edges. All four tasks form one antichain at the audited revision. The chosen sequence serializes the shared worker; it does not imply a code prerequisite. Each task owns its stated defect. S01 owns hash-boundary text. S02 owns command directory behavior. S03 owns runner outcome errors. S04 owns recovery work and retention. No task owns a second task's requirements. +## Keep CAS integration sequence + +Use `~/git/keep` as the source project. The local Keep checkout is `001ae2a`; its refreshed `origin/main` is `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. Inspect and pin the actual integration revision. Preserve the existing paused checkout. + +[Issue #722](https://github.com/flyingrobots/echo/issues/722) is the integration container. Its six milestones describe the same work as the six task documents below. Do not count the container as another executable PR. + +The [physical-content boundary](docs/architecture/echo-keep-physical-content-boundary.md) owns the accepted architecture. Keep supplies physical bytes and receipts. Echo retains content identity, WSC identity, causal history, semantic meaning, and authority. Keep receipts cannot replace Echo observations. + +Echo currently pins Rust 1.96.0. Keep requires Rust 1.96. The older Rust 1.90 statement in #722 is stale. A matching version number does not prove dependency, platform, or durability compatibility. + +Keep main exports `DurableStore` and fenced snapshots. Its crate documentation still states that production durable ingestion, garbage collection, and compaction are unimplemented. Thus durable reads and durable publication have different readiness. + +- [ ] [K01: Prove the Echo and Keep content identity bridge](tasks/K01.md) — [issue #759](https://github.com/flyingrobots/echo/issues/759) +- [ ] [K02: Add the Echo physical-content port and existing CAS adapters](tasks/K02.md) — [issue #760](https://github.com/flyingrobots/echo/issues/760) +- [ ] [K03: Add an experimental Keep ReferenceStore adapter](tasks/K03.md) — [issue #761](https://github.com/flyingrobots/echo/issues/761) +- [ ] [K04: Add a pinned-generation durable Keep read adapter](tasks/K04.md) +- [ ] [K05: Prove durable Echo and Keep publication reconciliation](tasks/K05.md) +- [ ] [K06: Prepare the migration and production adoption decision](tasks/K06.md) + +Execute K01, K02, K03, then K04. K05 requires Keep's missing durable publication capabilities and reconciliation anchors. K06 requires the complete crash, migration, and rollback evidence. + +Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. K01 supplies the identity law. K02 supplies the port. K03 supplies backend conformance. K04 supplies pinned-view receipt validation. K05 supplies durable reconciliation. + +No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. + +The combined proposed graph has ten task vertices and five edges. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External Keep publication prerequisites remain unresolved outside that graph. Tracker edges require reconciliation before execution. + +The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. + ## Verification and merge rules Use the existing `echo-read-runtime` worker and stable `/lease-target` cache. Use `/Users/Shared/git-locks/workstation.git` with `host/heavy-work` and `host/docker/echo-read-runtime/`. The monitor also inspects `echo-provider-builder`; reserve its key during validation. diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index b0ed01c2..0f5c44a3 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,6 +163,8 @@ when one exists. ## Writing and structure +Markdown frontmatter remains first. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates and repairs this position. + - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. - Use active voice when it clarifies ownership. diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index f3353e2c..09cdb8e8 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -125,6 +125,17 @@ check_valid_header() { i=1 fi + # Markdown metadata must remain first. Check the license immediately after + # a complete frontmatter block rather than moving it ahead of the metadata. + if [[ "$f" == *.md && "$first_line_in_file" == "---" ]]; then + i=1 + while [[ "$i" -lt "${#file_lines[@]}" && "${file_lines[i]}" != "---" ]]; do + i=$((i + 1)) + done + if [[ "$i" -ge "${#file_lines[@]}" ]]; then return 1; fi + i=$((i + 1)) + fi + # Extract the lines from expected_header_block local expected_lines=() # Use process substitution with IFS=$'\n' to split multi-line string into array elements @@ -171,18 +182,25 @@ strip_existing_headers() { # This effectively removes "bad" headers or "wrong license" headers. # We preserve shebangs because they typically don't match the pattern. - awk ' + local metadata_lines=0 + if [[ "$f" == *.md && "$(head -n 1 "$f")" == "---" ]]; then + metadata_lines=$(awk 'NR > 1 && $0 == "---" { print NR; exit }' "$f") + metadata_lines=${metadata_lines:-0} + fi + + awk -v header_start="$metadata_lines" ' BEGIN { header_block_active = 1; line_num = 0 } { line_num++; + if (line_num <= header_start) { print; next; } if (header_block_active) { # Once we pass line 15, we are out of the header block. - if (line_num > 15) { header_block_active = 0; } + if (line_num > header_start + 15) { header_block_active = 0; } # If it is not a SPDX/Copyright line, and it is not a shebang/xml declaration, # then we are likely past the header block. # This condition is crucial for `in_header_block` to become 0. - if (line_num > 1 && $0 !~ /^#!/ && $0 !~ /^\<\?xml/ && $0 !~ /SPDX-License-Identifier/ && $0 !~ /James Ross .* FLYING/) { + if (line_num > header_start + 1 && $0 !~ /^#!/ && $0 !~ /^\<\?xml/ && $0 !~ /SPDX-License-Identifier/ && $0 !~ /James Ross .* FLYING/) { header_block_active = 0; } @@ -214,8 +232,18 @@ insert_header() { local first_line first_line=$(head -n 1 "$f" || true) - # Logic to insert header AFTER shebang/xml declaration if present - if [[ "$first_line" =~ ^#! ]]; then + # Keep complete Markdown metadata ahead of the license in repair mode too. + local metadata_lines="" + if [[ "$f" == *.md && "$first_line" == "---" ]]; then + metadata_lines=$(awk 'NR > 1 && $0 == "---" { print NR; exit }' "$f") + fi + + # Logic to insert header AFTER metadata/shebang/xml declaration if present + if [[ -n "$metadata_lines" ]]; then + head -n "$metadata_lines" "$f" > "$temp_file" + echo "$header" >> "$temp_file" + tail -n "+$((metadata_lines + 1))" "$f" >> "$temp_file" + elif [[ "$first_line" =~ ^#! ]]; then echo "$first_line" > "$temp_file" echo "$header" >> "$temp_file" tail -n +2 "$f" >> "$temp_file" diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh new file mode 100755 index 00000000..338498a3 --- /dev/null +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# © James Ross Ω FLYING•ROBOTS +set -euo pipefail +root=$(git rev-parse --show-toplevel) +cd "$root" +checker=${1:-scripts/ensure_spdx.sh} +work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") +trap 'rm -rf "$work"' EXIT +cat > "$work/valid.md" <<'DOC' +--- +id: S01 +type: Feature +--- + + +# Task +DOC +bash "$checker" --check "$work/valid.md" +cat > "$work/missing.md" <<'DOC' +--- +id: S01 +--- +# Task +DOC +if bash "$checker" --check "$work/missing.md"; then + echo 'missing license passed' >&2 + exit 1 +fi +if bash "$checker" "$work/missing.md"; then + echo 'repair must report changed files' >&2 + exit 1 +else + test "$?" = 1 +fi +bash "$checker" --check "$work/missing.md" +test "$(head -n 1 "$work/missing.md")" = '---' +test "$(sed -n '2p' "$work/missing.md")" = 'id: S01' +sed 's/Apache-2.0 OR LicenseRef-MIND-UCAL-1.0/MIT/' "$work/valid.md" > "$work/wrong.md" +if bash "$checker" "$work/wrong.md"; then + echo 'repair must report changed files' >&2 + exit 1 +else + test "$?" = 1 +fi +cmp "$work/valid.md" "$work/wrong.md" +cat > "$work/unclosed.md" <<'DOC' +--- +id: S01 + + +DOC +if bash "$checker" --check "$work/unclosed.md"; then + echo 'unclosed frontmatter passed' >&2 + exit 1 +fi +printf '%s\n' 'PASS: valid metadata license, missing license refusal, unclosed metadata refusal' diff --git a/tasks/K01.md b/tasks/K01.md new file mode 100644 index 00000000..4c282285 --- /dev/null +++ b/tasks/K01.md @@ -0,0 +1,70 @@ +--- +id: K01 +type: Feature +title: Prove the Echo and Keep content identity bridge +issue: https://github.com/flyingrobots/echo/issues/759 +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [] +--- + + + +# Feature + +```text +Execute task K01 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Prove the Echo and Keep content identity bridge. Current Echo has no physical-content port or Keep adapter. + +## 3. Proposed Solution + +Read both exact identity laws. Compute both identities from one bounded source stream. Reconstruct through Keep. Verify the Echo hash and exact length again. Keep the binding private and versioned. + +## 4. Prerequisites + +- [ ] Recheck exact Echo and Keep identity source and supported toolchains. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +A golden-vector suite proves same-source and reconstructed-source agreement. A wrong identity or length refuses. The old Echo CAS remains the default. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Empty bytes, text, binary ramps, chunk boundaries, interrupted sources, substituted identities, and length mismatches. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K02.md b/tasks/K02.md new file mode 100644 index 00000000..80458e53 --- /dev/null +++ b/tasks/K02.md @@ -0,0 +1,70 @@ +--- +id: K02 +type: Feature +title: Add the Echo physical-content port and existing CAS adapters +issue: https://github.com/flyingrobots/echo/issues/760 +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [] +--- + + + +# Feature + +```text +Execute task K02 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Add the Echo physical-content port and existing CAS adapters. Current Echo has no physical-content port or Keep adapter. + +## 3. Proposed Solution + +Add an Echo-owned fallible complete-object port. Implement MemoryTier and DiskTier adapters. Keep types inside backend adapters. Require private staging and atomic output promotion. Report unsupported evidence explicitly. + +## 4. Prerequisites + +- [ ] Recheck the Echo physical-content boundary and existing CAS adapter contracts. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +Existing backends pass the same complete-object tests. No failed read exposes a partial object. Existing consumers retain compatible behavior. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Missing content, corrupted content, failed writes, failed promotion, resource refusal, and unchanged visible output after failure. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K03.md b/tasks/K03.md new file mode 100644 index 00000000..9093e79e --- /dev/null +++ b/tasks/K03.md @@ -0,0 +1,71 @@ +--- +id: K03 +type: Feature +title: Add an experimental Keep ReferenceStore adapter +issue: https://github.com/flyingrobots/echo/issues/761 +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [K01, K02] +--- + + + +# Feature + +```text +Execute task K03 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Add an experimental Keep ReferenceStore adapter. Current Echo has no physical-content port or Keep adapter. + +## 3. Proposed Solution + +Pin the inspected Keep revision. Add an optional Echo adapter over Keep ReferenceStore. Reuse the port conformance suite. Keep the adapter disabled by default and label its receipts non-durable. + +## 4. Prerequisites + +- [ ] Complete [K01](K01.md). Its contract supplies required behavior. +- [ ] Complete [K02](K02.md). Its contract supplies required behavior. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +Both existing CAS and Keep pass one backend-neutral suite. The adapter changes no Echo or WSC identity. No ReferenceStore receipt claims restart durability. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Byte equality, identity substitution, missing chunks, corrupt layouts, deterministic layout selection, capacity limits, and source interruption. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K04.md b/tasks/K04.md new file mode 100644 index 00000000..c6103263 --- /dev/null +++ b/tasks/K04.md @@ -0,0 +1,69 @@ +--- +id: K04 +type: Feature +title: Add a pinned-generation durable Keep read adapter +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [K03] +--- + + + +# Feature + +```text +Execute task K04 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Add a pinned-generation durable Keep read adapter. Current Echo has no physical-content port or Keep adapter. + +## 3. Proposed Solution + +Verify current Keep DurableStore and DurableView contracts at the selected revision. Add an optional adapter for an admitted immutable generation. Retain its evidence during the read. Reconstruct into bounded private staging and validate the full receipt. + +## 4. Prerequisites + +- [ ] Complete [K03](K03.md). Its contract supplies required behavior. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +The adapter names and retains the exact view. Every success binds the requested content and length. Unsupported platforms refuse. This read adapter does not claim durable ingestion. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Generation changes, reader fencing, unsupported platforms, stale retention roots, partial output, corrupt retained closure, cancellation, and allocation limits. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K05.md b/tasks/K05.md new file mode 100644 index 00000000..2c96b803 --- /dev/null +++ b/tasks/K05.md @@ -0,0 +1,69 @@ +--- +id: K05 +type: Feature +title: Prove durable Echo and Keep publication reconciliation +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [K04] +--- + + + +# Feature + +```text +Execute task K05 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Prove durable Echo and Keep publication reconciliation. Current Echo has no physical-content port or Keep adapter. + +## 3. Proposed Solution + +First verify that Keep exposes durable ingestion, publication, retention anchors, and operation lookup. If any capability is absent, record the blocker on #722. After those prerequisites exist, retain one operation identity. Commit the Echo request before the physical effect. Publish Keep content under a non-expiring anchor. Settle Echo before dependent work resumes. Finalize retention before anchor release. + +## 4. Prerequisites + +- [ ] Complete [K04](K04.md). Its contract supplies required behavior. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +The bounded crash matrix proves idempotent recovery. An authoritative Echo reference never names unavailable physical content. The default backend remains unchanged. Missing prerequisite capabilities block execution. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Process death before and after each request, publication, settlement, and retention boundary. Duplicate retries, unavailable Echo, corrupt evidence, and cancellation. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K06.md b/tasks/K06.md new file mode 100644 index 00000000..747c40bb --- /dev/null +++ b/tasks/K06.md @@ -0,0 +1,69 @@ +--- +id: K06 +type: Feature +title: Prepare the migration and production adoption decision +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [K05] +--- + + + +# Feature + +```text +Execute task K06 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Prepare the migration and production adoption decision. Current Echo has no physical-content port or Keep adapter. + +## 3. Proposed Solution + +Run bounded shadow comparison, backfill, and rollback rehearsals. Retain both content identities and explicit backend provenance. Present the exact platform, durability, retention, and compatibility evidence. Record the accepted decision in the canonical physical-content document before any production cutover. + +## 4. Prerequisites + +- [ ] Complete [K05](K05.md). Its contract supplies required behavior. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +The reviewer can assess the complete adoption evidence. The decision records the selected posture and unresolved limits. No production cutover occurs without an explicit accepted decision. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Partial migration, substituted binding, unavailable Keep, explicit migration fallback, rollback, and fallback removal. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/S01.md b/tasks/S01.md index 278e4ec8..5ca14a27 100644 --- a/tasks/S01.md +++ b/tasks/S01.md @@ -6,6 +6,8 @@ issue: https://github.com/flyingrobots/echo/issues/754 evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 depends_on: [] --- + + # Feature @@ -52,7 +54,7 @@ The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact ## 8. Test Plan -Golden: `Inspect the old and new API text. Run a focused warp-core test that creates two detached cells, compares roots, and checks distinct retained patches and commits.` +Golden: Inspect the old and new API text. Run a focused warp-core test that creates two detached cells, compares roots, and checks distinct retained patches and commits. Edges: Reachable nodes, detached nodes, descended instances, and duplicate no-mutation evidence. diff --git a/tasks/S02.md b/tasks/S02.md index bacd8f9a..b79c6b26 100644 --- a/tasks/S02.md +++ b/tasks/S02.md @@ -6,6 +6,8 @@ issue: https://github.com/flyingrobots/echo/issues/755 evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 depends_on: [] --- + + # Feature @@ -52,7 +54,7 @@ The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact ## 8. Test Plan -Golden: `cargo test -p xtask --test run_edict_operation. Add cases for a directory outside Git and a nested directory inside an unrelated repository.` +Golden: `cargo test -p xtask --test run_edict_operation`. Add cases for a directory outside Git and a nested directory inside an unrelated repository. Edges: Absolute paths, relative paths, nested directories, malformed inputs, and repository commands. diff --git a/tasks/S03.md b/tasks/S03.md index cc438a08..172b284c 100644 --- a/tasks/S03.md +++ b/tasks/S03.md @@ -6,6 +6,8 @@ issue: https://github.com/flyingrobots/echo/issues/756 evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 depends_on: [] --- + + # Feature @@ -52,7 +54,7 @@ The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact ## 8. Test Plan -Golden: `cargo test -p xtask --test run_edict_operation. Use a canonical input whose retained projection exceeds its declared output bound.` +Golden: `cargo test -p xtask --test run_edict_operation`. Use a canonical input whose retained projection exceeds its declared output bound. Edges: Missing outcome, projection refusal, committed outcome, and duplicate obstruction. diff --git a/tasks/S04.md b/tasks/S04.md index 46aee711..3feee0e3 100644 --- a/tasks/S04.md +++ b/tasks/S04.md @@ -6,6 +6,8 @@ issue: https://github.com/flyingrobots/echo/issues/757 evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 depends_on: [] --- + + # Feature @@ -44,7 +46,7 @@ Out: Application-specific APIs, provider schema changes, new hash versions, and ## 6. Acceptance Criteria -A multi-tick recovery witness counts applied replay patches and retained replay states. Counts grow with required history, not the sum of every prefix. Poisoned bases and reordered or incomplete outcomes still refuse. +A multi-tick recovery witness counts applied replay patches and retained replay states. Applied replay-patch work grows with required history, not the sum of every prefix. Full replay snapshots stay bounded by worldline count. The witness accounts for their graph and tick-history data. Poisoned bases and reordered or incomplete outcomes still refuse. ## 7. Definition of Done @@ -52,7 +54,7 @@ The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact ## 8. Test Plan -Golden: `cargo test -p warp-core --features native_rule_bootstrap,trusted_runtime,host_test --test executable_operation_pipeline_tests. Add deterministic work counters for consecutive ticks and mixed basis coordinates.` +Golden: `cargo test -p warp-core --features native_rule_bootstrap,trusted_runtime,host_test --test executable_operation_pipeline_tests`. Add deterministic work counters for consecutive ticks and mixed basis coordinates. Edges: Shared bases, stale bases, multiple worldlines, checkpoints, batches, conflicts, and corrupted retained evidence. From b4584a10af42f752b809128f131dda3bd7702517 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 05:01:42 -0700 Subject: [PATCH 03/27] docs: refine recovery and CAS solutions from independent feedback --- ROADMAP.md | 18 +++++++++++++++--- tasks/K01.md | 4 ++++ tasks/K04.md | 10 +++++++++- tasks/K05.md | 9 +++++++++ tasks/K06.md | 2 ++ tasks/S01.md | 2 +- tasks/S02.md | 2 +- tasks/S03.md | 6 +++--- tasks/S04.md | 15 ++++++++++++--- 9 files changed, 56 insertions(+), 12 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index c5072128..a8b8316a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -62,7 +62,7 @@ Use `~/git/keep` as the source project. The local Keep checkout is `001ae2a`; it The [physical-content boundary](docs/architecture/echo-keep-physical-content-boundary.md) owns the accepted architecture. Keep supplies physical bytes and receipts. Echo retains content identity, WSC identity, causal history, semantic meaning, and authority. Keep receipts cannot replace Echo observations. -Echo currently pins Rust 1.96.0. Keep requires Rust 1.96. The older Rust 1.90 statement in #722 is stale. A matching version number does not prove dependency, platform, or durability compatibility. +Echo selects Rust 1.96.0 for its toolchain and warp-core. The workspace default and echo-cas still declare Rust 1.90.0. Keep requires Rust 1.96. The older Rust 1.90 statement in #722 is stale. A matching version number does not prove dependency, platform, or durability compatibility. Keep main exports `DurableStore` and fenced snapshots. Its crate documentation still states that production durable ingestion, garbage collection, and compaction are unimplemented. Thus durable reads and durable publication have different readiness. @@ -73,16 +73,28 @@ Keep main exports `DurableStore` and fenced snapshots. Its crate documentation s - [ ] [K05: Prove durable Echo and Keep publication reconciliation](tasks/K05.md) - [ ] [K06: Prepare the migration and production adoption decision](tasks/K06.md) -Execute K01, K02, K03, then K04. K05 requires Keep's missing durable publication capabilities and reconciliation anchors. K06 requires the complete crash, migration, and rollback evidence. +Execute K01 and K02 as independent slices, then K03. Stop this run after the experimental adapter passes its conformance and review gates. + +K04–K06 are conditional follow-on work. K04 requires guarded storage that actually passes Keep's Linux ext4 admission. K05 is blocked on external Keep durable ingestion, operation lookup, and non-expiring retention anchors. K06 also requires crash, migration, rollback, and adoption evidence. Completing K03 does not unblock these requirements. Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. K01 supplies the identity law. K02 supplies the port. K03 supplies backend conformance. K04 supplies pinned-view receipt validation. K05 supplies durable reconciliation. No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. -The combined proposed graph has ten task vertices and five edges. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External Keep publication prerequisites remain unresolved outside that graph. Tracker edges require reconciliation before execution. +The combined proposed graph has ten task vertices and five edges. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. Tracker edges require reconciliation before execution. The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. +## Solution constraints from independent critique + +S03 reports bounded outcome categories. It distinguishes a missing outcome from an obstruction and avoids raw record dumps. + +S04 indexes exact obligations before replay. It sorts verification references within each worldline and keeps retained protocol order authoritative. Its two-sweep plan supports stale and cross-worldline bases. Replay counters bound patch applications; they do not prove linear elapsed time. + +K01 verifies two distinct identity laws on the same source and reconstructed bytes. Equal source bytes do not make Echo and Keep digest values equal. K02 preserves existing CAS defaults and package compatibility. + +The independent critique confirms the S01 contract repair and experimental K01–K03 boundary. Its Reader source is receipt `7e05db3c-49d0-45df-b81b-2f1a6a25a6f7`. Reconciliation details remain on PR #758; this plan contains the resulting requirements. + ## Verification and merge rules Use the existing `echo-read-runtime` worker and stable `/lease-target` cache. Use `/Users/Shared/git-locks/workstation.git` with `host/heavy-work` and `host/docker/echo-read-runtime/`. The monitor also inspects `echo-provider-builder`; reserve its key during validation. diff --git a/tasks/K01.md b/tasks/K01.md index 4c282285..2e6d7632 100644 --- a/tasks/K01.md +++ b/tasks/K01.md @@ -33,6 +33,10 @@ Prove the Echo and Keep content identity bridge. Current Echo has no physical-co Read both exact identity laws. Compute both identities from one bounded source stream. Reconstruct through Keep. Verify the Echo hash and exact length again. Keep the binding private and versioned. +Echo hashes raw content bytes with BLAKE3. Keep hashes its versioned domain, the content, and the exact length. Verify each identity against its own golden vectors. Do not require the two digests to equal each other. Reconstruct from Keep, then verify the Echo identity and length again. + +Use an isolated experimental adapter package with an explicit Rust 1.96 requirement. Keep the existing echo-cas Rust 1.90 package posture and default dependency graph intact. + ## 4. Prerequisites - [ ] Recheck exact Echo and Keep identity source and supported toolchains. diff --git a/tasks/K04.md b/tasks/K04.md index c6103263..8b1d2e53 100644 --- a/tasks/K04.md +++ b/tasks/K04.md @@ -6,6 +6,8 @@ tracking_issue: https://github.com/flyingrobots/echo/issues/722 echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 depends_on: [K03] +execution_scope: conditional_follow_on +external_prerequisites: [admitted_linux_ext4_storage, continuous_storage_guard] --- @@ -30,7 +32,13 @@ Add a pinned-generation durable Keep read adapter. Current Echo has no physical- ## 3. Proposed Solution -Verify current Keep DurableStore and DurableView contracts at the selected revision. Add an optional adapter for an admitted immutable generation. Retain its evidence during the read. Reconstruct into bounded private staging and validate the full receipt. +Verify current Keep DurableStore and DurableView contracts at the selected revision. This task is outside the current experimental-adapter execution scope. + +First prove Keep admission on the exact backing filesystem. The durable profile requires Linux ext4 and its stated directory, mount, and synchronization conditions. A Linux container over overlayfs does not establish admission. If guarded admitted storage is unavailable, record the blocker. Do not forge platform evidence or create an unguarded filesystem. + +Run required tests in Docker. Do not use host test fallbacks. Unsupported-platform checks require a permitted Docker or cross-target route that preserves the execution policy. + + Add an optional adapter for an admitted immutable generation. Retain its evidence during the read. Reconstruct into bounded private staging and validate the full receipt. ## 4. Prerequisites diff --git a/tasks/K05.md b/tasks/K05.md index 2c96b803..e2e46495 100644 --- a/tasks/K05.md +++ b/tasks/K05.md @@ -6,6 +6,9 @@ tracking_issue: https://github.com/flyingrobots/echo/issues/722 echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 depends_on: [K04] +execution_scope: conditional_follow_on +readiness: blocked_external +external_prerequisites: [keep_durable_ingestion, durable_operation_lookup, non_expiring_retention_anchor, admitted_guarded_storage] --- @@ -34,6 +37,12 @@ First verify that Keep exposes durable ingestion, publication, retention anchors ## 4. Prerequisites +- [ ] Keep exposes and verifies durable ingestion for the selected profile. +- [ ] Keep exposes durable operation lookup and a non-expiring reconciliation anchor. +- [ ] The shared guard accounts for the admitted backing store and every output. + +These external requirements are unresolved at Keep `3165890e`. No external issue ID is claimed. Issue #722 must record the accepted prerequisites and actual tracker links before this task can execute. + - [ ] Complete [K04](K04.md). Its contract supplies required behavior. Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. diff --git a/tasks/K06.md b/tasks/K06.md index 747c40bb..a54ac852 100644 --- a/tasks/K06.md +++ b/tasks/K06.md @@ -6,6 +6,8 @@ tracking_issue: https://github.com/flyingrobots/echo/issues/722 echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 depends_on: [K05] +execution_scope: conditional_follow_on +readiness: blocked_external --- diff --git a/tasks/S01.md b/tasks/S01.md index 5ca14a27..9e614c68 100644 --- a/tasks/S01.md +++ b/tasks/S01.md @@ -46,7 +46,7 @@ Out: Application-specific APIs, provider schema changes, new hash versions, and ## 6. Acceptance Criteria -The API states the reachable-state boundary. The operation guide states that a new detached cell can leave the state root unchanged. A witness binds the write through its patch and commit. +The API states the reachable-state boundary. `docs/architecture/application-contract-hosting.md` and `docs/topics/WAL.md` state that a detached cell can leave the state root unchanged. A witness binds the write through its patch and commit. ## 7. Definition of Done diff --git a/tasks/S02.md b/tasks/S02.md index b79c6b26..2501cd87 100644 --- a/tasks/S02.md +++ b/tasks/S02.md @@ -29,7 +29,7 @@ main() resolves a Git root and changes the current directory before it parses th ## 3. Proposed Solution -Parse the command first. Keep the caller directory for run-edict-operation. Resolve the Git root only for commands that require repository paths. +Parse the command first. Exempt run-edict-operation from both Git-root discovery and the directory change. Keep both steps for repository commands. ## 4. Prerequisites diff --git a/tasks/S03.md b/tasks/S03.md index 172b284c..e871dbed 100644 --- a/tasks/S03.md +++ b/tasks/S03.md @@ -29,7 +29,7 @@ The runner discards a non-committed Action outcome. It reports only that the sch ## 3. Proposed Solution -Include the typed outcome in the runner error. Distinguish an absent outcome from an obstruction. Keep retained obstruction encoding unchanged. +Match the first Action outcome and the unexpected duplicate outcome explicitly. Use the same bounded formatter for both paths. Report an obstruction through its fieldless kind. Report footprint conflict with a fixed category and a bounded count. Report an absent outcome separately. Do not print full Debug records or invocation data. Keep retained obstruction encoding unchanged. ## 4. Prerequisites @@ -46,7 +46,7 @@ Out: Application-specific APIs, provider schema changes, new hash versions, and ## 6. Acceptance Criteria -An output-bound refusal names ResultProjectionInvalid. The runner emits no successful report. Existing durable obstruction and duplicate checks still pass. +An output-bound refusal names ResultProjectionInvalid. Each outcome error stays within 256 UTF-8 bytes and contains no invocation data. The runner emits no successful report. Existing durable obstruction and duplicate checks still pass. ## 7. Definition of Done @@ -56,7 +56,7 @@ The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact Golden: `cargo test -p xtask --test run_edict_operation`. Use a canonical input whose retained projection exceeds its declared output bound. -Edges: Missing outcome, projection refusal, committed outcome, and duplicate obstruction. +Edges: Missing outcome, projection refusal, footprint conflict, committed outcome, and duplicate obstruction. Check the message length and absence of raw record fields. Known failure modes: The runner discards a non-committed Action outcome. It reports only that the scheduler did not publish a committed outcome. ResultProjectionInvalid therefore disappears from the caller error. diff --git a/tasks/S04.md b/tasks/S04.md index 3feee0e3..2d954cce 100644 --- a/tasks/S04.md +++ b/tasks/S04.md @@ -29,7 +29,16 @@ Recovery caches a full WorldlineState for every distinct basis tick. Each cache ## 3. Proposed Solution -Process required coordinates in causal order with bounded replay state. Reuse verified prefixes. Preserve all basis, patch, receipt, decision, conflict, and initial-boundary checks. +Index verification obligations by exact coordinate without changing retained protocol order. Use bounded replay cursors and the existing replay validation primitives. Preserve all basis, patch, receipt, decision, conflict, and initial-boundary checks. + +1. Validate record identities and order under the existing laws. Index receipt and Action basis obligations by worldline and tick. Index composite Ticks separately by their parent coordinate. Store references and compact preparation evidence, not full states. +2. Visit basis obligations in ascending tick order within each worldline. Validate the initial boundary. Restore only an independently checked initial state or checkpoint at or before the first needed coordinate. Advance one cursor through retained patches. Verify the root and commit at every step. Validate all obligations at each selected basis. Retain exact preparation evidence for later Tick reconstruction. +3. After every basis obligation is checked, visit composite Tick parents in ascending order within each worldline. Reuse or restart the cursor through the same checked replay primitives. Preserve scheduler ordering, conflicts, member indices, obstructions, result identities, and patch commitments. This second sweep permits an Action basis on another worldline without premature Tick reconstruction. +4. Retain at most one replay cursor per needed worldline and one transient Tick simulation state. Discard the simulation after its checks. Publish no live runtime change before validation succeeds. Add separate counters for prefix initializations, replayed patches, peak cursor states, and peak retained graph/history data. + +One sweep is an acceptable alternative only after a witness proves that every preparation is available before its Tick validation. Never re-evaluate BasisChanged as a committed Action. Sorting verification references must not repair a malformed retained order. + +Bound replay-patch applications by two verified history sweeps per parent-state validation call. Count other activation work separately. This bound describes replay applications, not total CPU time. State hashing and Tick simulation still have graph-dependent cost. Preserve the legacy test helper's stated meaning or add a separate structured counter helper. Do not silently redefine its result. ## 4. Prerequisites @@ -46,7 +55,7 @@ Out: Application-specific APIs, provider schema changes, new hash versions, and ## 6. Acceptance Criteria -A multi-tick recovery witness counts applied replay patches and retained replay states. Applied replay-patch work grows with required history, not the sum of every prefix. Full replay snapshots stay bounded by worldline count. The witness accounts for their graph and tick-history data. Poisoned bases and reordered or incomplete outcomes still refuse. +A multi-tick recovery witness counts applied replay patches and retained replay states. Applied replay-patch work stays within two history sweeps. At most one cursor exists per needed worldline, plus one transient Tick simulation state. The witness bounds retained graph and tick-history data, not just object count. Poisoned bases and reordered or incomplete outcomes still refuse. ## 7. Definition of Done @@ -56,7 +65,7 @@ The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact Golden: `cargo test -p warp-core --features native_rule_bootstrap,trusted_runtime,host_test --test executable_operation_pipeline_tests`. Add deterministic work counters for consecutive ticks and mixed basis coordinates. -Edges: Shared bases, stale bases, multiple worldlines, checkpoints, batches, conflicts, and corrupted retained evidence. +Edges: Shared bases, delayed stale bases, non-monotonic retained basis order, cross-worldline bases, multiple heads, checkpoints, batches, conflicts, and corrupted retained evidence. Compare accepted and refused postures with the old validator. Known failure modes: Recovery caches a full WorldlineState for every distinct basis tick. Each cache miss replays from the initial boundary when no checkpoint exists. Consecutive bases repeat all earlier patches and retain growing graph and tick-history copies. From 861a5c99a3bf7670c59df422e2bc623370a59a0e Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 05:26:12 -0700 Subject: [PATCH 04/27] docs: distinguish initial and remaining integration tasks --- ROADMAP.md | 2 +- tasks/K04.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 5205979e..2e969815 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -83,7 +83,7 @@ Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. -The combined proposed graph has ten task vertices and five edges. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. Tracker edges require reconciliation before execution. +The initial proposed graph has ten task vertices and five internal edges. S01 is complete. The remaining graph has nine task vertices and five internal edges; its first layer is `{S02,S03,S04,K01,K02}`. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. Tracker edges require reconciliation before execution. The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. diff --git a/tasks/K04.md b/tasks/K04.md index 8b1d2e53..c9bcb1c9 100644 --- a/tasks/K04.md +++ b/tasks/K04.md @@ -7,7 +7,7 @@ echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 depends_on: [K03] execution_scope: conditional_follow_on -external_prerequisites: [admitted_linux_ext4_storage, continuous_storage_guard] +external_prerequisites: [admitted_guarded_storage] --- From 93d5774be58c8bf986618b25800e1f4817c219bf Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 05:27:20 -0700 Subject: [PATCH 05/27] docs: remove leftover changelog merge marker --- CHANGELOG.md | 1 - 1 file changed, 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 761df4e4..a9ef6a56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,6 @@ - The SPDX checker preserves Markdown frontmatter when it checks or repairs license headers. -||||||| a93e9d82 - `WorldlineState::state_root` documentation now states its reachable-state boundary. Detached create-if-absent writes remain bound by patch and commit identities. Hash bytes are unchanged. - `ReadView::at` validates the aperture and derives the selected current frontier From a7027d0e9e374f28c2c4dffa891262a9bdb6a336 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 05:54:46 -0700 Subject: [PATCH 06/27] fix: preserve Markdown metadata and body during license repair --- .github/workflows/spdx-header-check.yml | 3 + AGENTS.md | 5 ++ docs/DOCUMENTATION_STANDARDS.md | 2 +- scripts/ensure_spdx.sh | 86 ++++++++++++++++++++----- scripts/tests/spdx_frontmatter_test.sh | 85 +++++++++++++++++++++++- 5 files changed, 163 insertions(+), 18 deletions(-) diff --git a/.github/workflows/spdx-header-check.yml b/.github/workflows/spdx-header-check.yml index 22c01833..eadaf3e5 100644 --- a/.github/workflows/spdx-header-check.yml +++ b/.github/workflows/spdx-header-check.yml @@ -26,6 +26,9 @@ jobs: chmod +x check_spdx.sh fi + - name: Verify Markdown license preservation + run: bash scripts/tests/spdx_frontmatter_test.sh + - name: Run SPDX check run: | if [ -f "scripts/check_spdx.sh" ]; then diff --git a/AGENTS.md b/AGENTS.md index f47daf32..7ba7fe2f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,6 +83,11 @@ issue or pull request. Record a durable decision in the named current document that owns its concept, and state supersession, refinement, dependency, and related-decision edges explicitly. Do not allocate a new numbered ADR. +The user-requested Echo study-feedback `ROADMAP.md` and linked `tasks/` cards +are a scoped exception to the checked-in-plan restriction. They project the +requested work; GitHub still owns issue status and accepted dependencies. +This exception does not authorize production Keep adoption or other backlogs. + When recovering context, read the relevant canonical topic/spec/invariant and architecture document, follow any explicit links into the historical ADR archive, then inspect the current GitHub issue or pull request, `git log -n 5`, diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index 0f5c44a3..632ea161 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -126,7 +126,7 @@ accepted contracts. GitHub owns change-local plans and status. Git history owns the exact old text. Do not check in backlogs, cycle packets, retrospectives, review transcripts, -status ledgers, or roadmap checklists. A short checked-in redirect may remain +status ledgers, or roadmap checklists. The user-requested Echo study-feedback ROADMAP and linked task cards are a scoped planning exception. GitHub remains their status and accepted-dependency authority. A short checked-in redirect may remain when an old stable path must route readers to its current owner. Historical reasoning must not masquerade as current behavior. Mark retained diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index 09cdb8e8..31d72726 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -107,6 +107,30 @@ get_header_content() { esac } +# Only complete HTML license/copyright comments belong to a Markdown header. +MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' +MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' + +markdown_metadata_bounds() { + awk -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + BEGIN { prefix = 1; start = 0; printed = 0; saw_header = 0 } + { + if (start > 0) { + if ($0 == "---") { print start, NR; printed = 1; exit } + next + } + if (NR == 1 && $0 == "---") { start = NR; next } + if (prefix) { + if ($0 ~ license_re || $0 ~ copyright_re) { saw_header = 1; next } + if ($0 ~ /^[[:space:]]*$/) { next } + if (saw_header && $0 == "---") { start = NR; next } + prefix = 0 + } + } + END { if (!printed) print start, 0 } + ' "$1" +} + check_valid_header() { local f="$1" local expected_header_block="$2" # This is a multi-line string like "# SPDX...\n# ©..." @@ -125,15 +149,14 @@ check_valid_header() { i=1 fi - # Markdown metadata must remain first. Check the license immediately after - # a complete frontmatter block rather than moving it ahead of the metadata. - if [[ "$f" == *.md && "$first_line_in_file" == "---" ]]; then - i=1 - while [[ "$i" -lt "${#file_lines[@]}" && "${file_lines[i]}" != "---" ]]; do - i=$((i + 1)) - done - if [[ "$i" -ge "${#file_lines[@]}" ]]; then return 1; fi - i=$((i + 1)) + if [[ "$f" == *.md ]]; then + local metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" + if [[ "$metadata_start" -gt 0 ]]; then + # A header-first metadata block must move to the start in repair mode. + if [[ "$metadata_start" -ne 1 || "$metadata_end" -eq 0 ]]; then return 1; fi + i=$metadata_end + fi fi # Extract the lines from expected_header_block @@ -178,17 +201,34 @@ strip_existing_headers() { local temp_file temp_file=$(mktemp) + if [[ "$f" == *.md ]]; then + local metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" + awk -v metadata_start="$metadata_start" -v metadata_end="$metadata_end" \ + -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + BEGIN { header_active = 1 } + { + # The bounds helper admits only header comments and whitespace here. + if (metadata_start > 1 && NR < metadata_start) next + if (metadata_start > 0 && NR <= metadata_end) { print; next } + if (header_active) { + if (NR > metadata_end + 15) header_active = 0 + else if ($0 ~ license_re || $0 ~ copyright_re) next + else if ($0 !~ /^[[:space:]]*$/) header_active = 0 + } + print + } + ' "$f" > "$temp_file" + cat "$temp_file" > "$f" + rm "$temp_file" + return + fi + # Use AWK to filter out lines in the first 15 lines that match SPDX/Copyright patterns. # This effectively removes "bad" headers or "wrong license" headers. # We preserve shebangs because they typically don't match the pattern. - local metadata_lines=0 - if [[ "$f" == *.md && "$(head -n 1 "$f")" == "---" ]]; then - metadata_lines=$(awk 'NR > 1 && $0 == "---" { print NR; exit }' "$f") - metadata_lines=${metadata_lines:-0} - fi - - awk -v header_start="$metadata_lines" ' + awk -v header_start=0 ' BEGIN { header_block_active = 1; line_num = 0 } { line_num++; @@ -271,6 +311,16 @@ process_file() { local expected_header_block expected_header_block=$(get_header_content "$f" "$style") + if [[ "$f" == *.md ]]; then + local metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" + if [[ "$metadata_start" -gt 0 && "$metadata_end" -eq 0 ]]; then + echo "[FAIL] Cannot process Markdown with unclosed frontmatter: $f" + FAILED_COUNT=$((FAILED_COUNT + 1)) + return + fi + fi + if check_valid_header "$f" "$expected_header_block"; then return 0 # Header is already perfect, nothing to do. fi @@ -325,6 +375,10 @@ if [[ "$CHECK_MODE" -eq 1 ]]; then exit 1 fi else + if [[ "$FAILED_COUNT" -gt 0 ]]; then + echo "SPDX Repair Failed: $FAILED_COUNT files could not be repaired." + exit 1 + fi if [[ "$MODIFIED_COUNT" -gt 0 ]]; then echo "-------------------------------------------------------" echo "Repaired SPDX headers in $MODIFIED_COUNT files." diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 338498a3..60411a30 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -5,6 +5,10 @@ set -euo pipefail root=$(git rev-parse --show-toplevel) cd "$root" checker=${1:-scripts/ensure_spdx.sh} +case "${SPDX_CASE:-all}" in + all|unclosed|header_first|displaced|prose) ;; + *) echo 'unknown SPDX regression case' >&2; exit 2 ;; +esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") trap 'rm -rf "$work"' EXIT cat > "$work/valid.md" <<'DOC' @@ -54,4 +58,83 @@ if bash "$checker" --check "$work/unclosed.md"; then echo 'unclosed frontmatter passed' >&2 exit 1 fi -printf '%s\n' 'PASS: valid metadata license, missing license refusal, unclosed metadata refusal' +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == unclosed ]]; then +cp "$work/unclosed.md" "$work/unclosed.original" +if bash "$checker" "$work/unclosed.md"; then + echo 'unclosed repair passed' >&2 + exit 1 +else + test "$?" = 1 +fi +cmp "$work/unclosed.original" "$work/unclosed.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == header_first ]]; then +cat > "$work/header-first.md" <<'DOC' + + + +--- +id: S01 +type: Feature +--- +# Task +DOC +if bash "$checker" --check "$work/header-first.md"; then + echo 'displaced metadata passed' >&2 + exit 1 +fi +if bash "$checker" "$work/header-first.md"; then + echo 'metadata relocation must report a change' >&2 + exit 1 +else + test "$?" = 1 +fi +bash "$checker" --check "$work/header-first.md" +head -n 4 "$work/valid.md" > "$work/metadata.expected" +head -n 4 "$work/header-first.md" > "$work/metadata.actual" +cmp "$work/metadata.expected" "$work/metadata.actual" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == displaced ]]; then +cat > "$work/displaced.md" <<'DOC' +--- +id: S01 +--- + + + + +# Task +DOC +if bash "$checker" "$work/displaced.md"; then + echo 'displaced license must report a change' >&2 + exit 1 +else + test "$?" = 1 +fi +bash "$checker" --check "$work/displaced.md" +test "$(grep -c '^[[:space:]]*$' +MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' markdown_metadata_bounds() { awk -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' - BEGIN { prefix = 1; start = 0; printed = 0; saw_header = 0 } - { - if (start > 0) { - if ($0 == "---") { print start, NR; printed = 1; exit } - next + { lines[NR] = $0 } + END { + start = 0 + if (lines[1] == "---") start = 1 + else { + saw_header = 0 + for (i = 1; i <= NR; i++) { + if (lines[i] ~ license_re || lines[i] ~ copyright_re) { saw_header = 1; continue } + if (lines[i] ~ /^[[:space:]]*$/) continue + if (saw_header && lines[i] == "---") start = i + break + } + } + if (!start) { print 0, 0; exit } + # A thematic break is body content. Task frontmatter begins with a map. + first = start + 1 + while (first <= NR && (lines[first] ~ /^[[:space:]]*$/ || lines[first] ~ /^[[:space:]]*#/)) first++ + if (first > NR || lines[first] !~ /^[[:space:]]*[[:alpha:]_][[:alnum:]_.-]*:([[:space:]]|$)/) { + print 0, 0; exit } - if (NR == 1 && $0 == "---") { start = NR; next } - if (prefix) { - if ($0 ~ license_re || $0 ~ copyright_re) { saw_header = 1; next } - if ($0 ~ /^[[:space:]]*$/) { next } - if (saw_header && $0 == "---") { start = NR; next } - prefix = 0 + for (i = start + 1; i <= NR; i++) { + if (lines[i] == "---") { print start, i; exit } } + print start, 0 } - END { if (!printed) print start, 0 } ' "$1" } diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 60411a30..e940d20f 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -6,7 +6,7 @@ root=$(git rev-parse --show-toplevel) cd "$root" checker=${1:-scripts/ensure_spdx.sh} case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -137,4 +137,39 @@ for spacer in 0 1; do bash "$checker" --check "$work/prose-$spacer.md" done fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == thematic ]]; then +for separator_count in 1 2; do + { + printf '%s\n' '' + printf '%s\n' '' + printf '\n---\n# Body heading\nA normal paragraph.\n' + if [[ "$separator_count" = 2 ]]; then printf '\n---\nMore body.\n'; fi + } > "$work/thematic-$separator_count.md" + cp "$work/thematic-$separator_count.md" "$work/thematic-$separator_count.original" + bash "$checker" --check "$work/thematic-$separator_count.md" + bash "$checker" "$work/thematic-$separator_count.md" + cmp "$work/thematic-$separator_count.original" "$work/thematic-$separator_count.md" +done +printf '%s\n' '---' '# Body heading' 'A normal paragraph.' > "$work/unlicensed-break.md" +cp "$work/unlicensed-break.md" "$work/unlicensed-break.original" +if bash "$checker" --check "$work/unlicensed-break.md"; then exit 1; fi +if bash "$checker" "$work/unlicensed-break.md"; then exit 1; else test "$?" = 1; fi +tail -n +3 "$work/unlicensed-break.md" > "$work/unlicensed-break.body" +cmp "$work/unlicensed-break.original" "$work/unlicensed-break.body" +bash "$checker" --check "$work/unlicensed-break.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == malformed ]]; then +cat > "$work/malformed-comment.md" <<'DOC' +--- +id: S01 +--- + + +# Task +DOC +if bash "$checker" "$work/malformed-comment.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/malformed-comment.md" +test "$(grep -c 'SPDX-License-Identifier' "$work/malformed-comment.md")" = 1 +grep -qx '# Task' "$work/malformed-comment.md" +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' From 6528fd176e635df92dc4e97a5ba604edb0e51bc2 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 07:30:34 -0700 Subject: [PATCH 09/27] docs: reconcile landed diagnostics and recorded Keep prerequisites --- ROADMAP.md | 4 ++-- tasks/K03.md | 2 +- tasks/S03.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index e88d9ca6..09a25deb 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -47,7 +47,7 @@ Code paths are relative to this repository, except the explicitly named hello-ec - [x] [S01: Describe the reachable-state boundary of WorldlineState::state_root](tasks/S01.md) — [issue #754](https://github.com/flyingrobots/echo/issues/754) - [x] [S02: Run xtask run-edict-operation without a Git checkout or directory change](tasks/S02.md) — [issue #755](https://github.com/flyingrobots/echo/issues/755) -- [ ] [S03: Report the typed Action obstruction when the operation runner cannot commit](tasks/S03.md) — [issue #756](https://github.com/flyingrobots/echo/issues/756) +- [x] [S03: Report the typed Action obstruction when the operation runner cannot commit](tasks/S03.md) — [issue #756](https://github.com/flyingrobots/echo/issues/756) - [ ] [S04: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery](tasks/S04.md) — [issue #757](https://github.com/flyingrobots/echo/issues/757) Execute S01, S02, S03, then S04. Each issue produces one independently mergeable PR. This order puts smaller contract and runner repairs before recovery work. @@ -83,7 +83,7 @@ Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. -The initial proposed graph has ten task vertices and five internal edges. S01 and S02 are complete. The remaining graph has eight task vertices and five internal edges; its first layer is `{S03,S04,K01,K02}`. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. Tracker edges require reconciliation before execution. +The initial proposed graph has ten task vertices and five internal edges. S01, S02, and S03 are complete. The remaining graph has seven task vertices and five internal edges; its first layer is `{S04,K01,K02}`. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. diff --git a/tasks/K03.md b/tasks/K03.md index 9093e79e..1ae786c1 100644 --- a/tasks/K03.md +++ b/tasks/K03.md @@ -38,7 +38,7 @@ Pin the inspected Keep revision. Add an optional Echo adapter over Keep Referenc - [ ] Complete [K01](K01.md). Its contract supplies required behavior. - [ ] Complete [K02](K02.md). Its contract supplies required behavior. -Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. +Recheck external capability prerequisites before execution. Record accepted edges on #722. GitHub records #759 and #760 as blockers of #761. Recheck the accepted edges before execution. ## 5. Scope diff --git a/tasks/S03.md b/tasks/S03.md index e871dbed..5428096f 100644 --- a/tasks/S03.md +++ b/tasks/S03.md @@ -46,7 +46,7 @@ Out: Application-specific APIs, provider schema changes, new hash versions, and ## 6. Acceptance Criteria -An output-bound refusal names ResultProjectionInvalid. Each outcome error stays within 256 UTF-8 bytes and contains no invocation data. The runner emits no successful report. Existing durable obstruction and duplicate checks still pass. +An output-bound refusal names ResultProjectionInvalid. Each outcome summary stays within 256 UTF-8 bytes and contains no invocation data. The runner emits no successful report. Existing durable obstruction and duplicate checks still pass. ## 7. Definition of Done From 7530a04c5f1d1dd43e53b764de6dec6fc0b2eff9 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 08:50:54 -0700 Subject: [PATCH 10/27] fix(tooling): preserve licensed prose and replace malformed SPDX comments --- scripts/ensure_spdx.sh | 14 +++++++++++--- scripts/tests/spdx_frontmatter_test.sh | 26 ++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 3 deletions(-) diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index 9200771c..a76703f0 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -108,7 +108,7 @@ get_header_content() { } # Only complete HTML license/copyright comments belong to a Markdown header. -MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' +MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' markdown_metadata_bounds() { @@ -133,10 +133,18 @@ markdown_metadata_bounds() { if (first > NR || lines[first] !~ /^[[:space:]]*[[:alpha:]_][[:alnum:]_.-]*:([[:space:]]|$)/) { print 0, 0; exit } + finish = 0 + has_id = 0 + has_type = 0 for (i = start + 1; i <= NR; i++) { - if (lines[i] == "---") { print start, i; exit } + if (lines[i] == "---") { finish = i; break } + if (lines[i] ~ /^[[:space:]]*id:[[:space:]]*[^[:space:]]/) has_id = 1 + if (lines[i] ~ /^[[:space:]]*type:[[:space:]]*(Feature|Bug|Decision|Research|Investigation|Spike)[[:space:]]*$/) has_type = 1 } - print start, 0 + # Only the explicit task-card schema authorizes automatic relocation. + # Generic body prose after a licensed header is never moved on key: alone. + if (start > 1 && !(has_id && has_type)) { print 0, 0; exit } + print start, finish } ' "$1" } diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index e940d20f..cd45a40f 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -157,6 +157,20 @@ if bash "$checker" "$work/unlicensed-break.md"; then exit 1; else test "$?" = 1; tail -n +3 "$work/unlicensed-break.md" > "$work/unlicensed-break.body" cmp "$work/unlicensed-break.original" "$work/unlicensed-break.body" bash "$checker" --check "$work/unlicensed-break.md" +cat > "$work/note-body.md" <<'DOC' + + + +--- +Note: this section explains metadata. + +A normal body paragraph. +--- +DOC +cp "$work/note-body.md" "$work/note-body.original" +bash "$checker" --check "$work/note-body.md" +bash "$checker" "$work/note-body.md" +cmp "$work/note-body.original" "$work/note-body.md" fi if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == malformed ]]; then cat > "$work/malformed-comment.md" <<'DOC' @@ -171,5 +185,17 @@ if bash "$checker" "$work/malformed-comment.md"; then exit 1; else test "$?" = 1 bash "$checker" --check "$work/malformed-comment.md" test "$(grep -c 'SPDX-License-Identifier' "$work/malformed-comment.md")" = 1 grep -qx '# Task' "$work/malformed-comment.md" +cat > "$work/equals-comment.md" <<'DOC' +--- +id: S01 +--- + + +# Task +DOC +if bash "$checker" "$work/equals-comment.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/equals-comment.md" +test "$(grep -c 'SPDX-License-Identifier' "$work/equals-comment.md")" = 1 +grep -qx '# Task' "$work/equals-comment.md" fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' From f8dd7d54f90cca2e85358dbc30cd78f4d4e4e52f Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 09:05:57 -0700 Subject: [PATCH 11/27] docs: bound automatic frontmatter relocation to task metadata --- docs/DOCUMENTATION_STANDARDS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index 632ea161..3d7c8a97 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,7 +163,7 @@ when one exists. ## Writing and structure -Markdown frontmatter remains first. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates and repairs this position. +Markdown frontmatter remains first. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with a nonempty `id` and a recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. From c9dc5f150932e70225898198cc798ef47373e050 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 09:07:00 -0700 Subject: [PATCH 12/27] docs: record merged recovery repair and remaining Keep tasks --- ROADMAP.md | 4 ++-- tasks/S04.md | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 09a25deb..f1154f7e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -48,7 +48,7 @@ Code paths are relative to this repository, except the explicitly named hello-ec - [x] [S01: Describe the reachable-state boundary of WorldlineState::state_root](tasks/S01.md) — [issue #754](https://github.com/flyingrobots/echo/issues/754) - [x] [S02: Run xtask run-edict-operation without a Git checkout or directory change](tasks/S02.md) — [issue #755](https://github.com/flyingrobots/echo/issues/755) - [x] [S03: Report the typed Action obstruction when the operation runner cannot commit](tasks/S03.md) — [issue #756](https://github.com/flyingrobots/echo/issues/756) -- [ ] [S04: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery](tasks/S04.md) — [issue #757](https://github.com/flyingrobots/echo/issues/757) +- [x] [S04: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery](tasks/S04.md) — [issue #757](https://github.com/flyingrobots/echo/issues/757) Execute S01, S02, S03, then S04. Each issue produces one independently mergeable PR. This order puts smaller contract and runner repairs before recovery work. @@ -83,7 +83,7 @@ Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. -The initial proposed graph has ten task vertices and five internal edges. S01, S02, and S03 are complete. The remaining graph has seven task vertices and five internal edges; its first layer is `{S04,K01,K02}`. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. +The initial proposed graph has ten task vertices and five internal edges. S01, S02, S03, and S04 are complete. The remaining graph has six task vertices and five internal edges; its first layer is `{K01,K02}`. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. diff --git a/tasks/S04.md b/tasks/S04.md index 2d954cce..605f21d7 100644 --- a/tasks/S04.md +++ b/tasks/S04.md @@ -3,6 +3,9 @@ id: S04 type: Feature title: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery issue: https://github.com/flyingrobots/echo/issues/757 +status: complete +pr: https://github.com/flyingrobots/echo/pull/767 +integration_commit: 7dde48b223ed105c13a1b70afb6a1a07d1dc308e evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 depends_on: [] --- From 6ad06c4a7297010e7bc5d493ed899982ec675acf Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 09:29:28 -0700 Subject: [PATCH 13/27] fix(tooling): preserve indented examples and recognize quoted metadata keys --- scripts/ensure_spdx.sh | 24 +++++++++---- scripts/tests/spdx_frontmatter_test.sh | 49 +++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 8 deletions(-) diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index a76703f0..b2c3bcee 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -108,11 +108,17 @@ get_header_content() { } # Only complete HTML license/copyright comments belong to a Markdown header. -MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' -MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*[[:space:]]*$' +MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*$' +MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*$' markdown_metadata_bounds() { awk -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + function map_key(line, quote) { + quote = sprintf("%c", 39) + return line ~ /^[[:space:]]*[[:alpha:]_][[:alnum:]_.-]*:([[:space:]]|$)/ || + line ~ /^[[:space:]]*".*"[[:space:]]*:([[:space:]]|$)/ || + line ~ ("^[[:space:]]*" quote ".*" quote "[[:space:]]*:([[:space:]]|$)") + } { lines[NR] = $0 } END { start = 0 @@ -130,7 +136,7 @@ markdown_metadata_bounds() { # A thematic break is body content. Task frontmatter begins with a map. first = start + 1 while (first <= NR && (lines[first] ~ /^[[:space:]]*$/ || lines[first] ~ /^[[:space:]]*#/)) first++ - if (first > NR || lines[first] !~ /^[[:space:]]*[[:alpha:]_][[:alnum:]_.-]*:([[:space:]]|$)/) { + if (first > NR || !map_key(lines[first])) { print 0, 0; exit } finish = 0 @@ -138,8 +144,8 @@ markdown_metadata_bounds() { has_type = 0 for (i = start + 1; i <= NR; i++) { if (lines[i] == "---") { finish = i; break } - if (lines[i] ~ /^[[:space:]]*id:[[:space:]]*[^[:space:]]/) has_id = 1 - if (lines[i] ~ /^[[:space:]]*type:[[:space:]]*(Feature|Bug|Decision|Research|Investigation|Spike)[[:space:]]*$/) has_type = 1 + if (lines[i] ~ /^id:[[:space:]]*[^[:space:]]/) has_id = 1 + if (lines[i] ~ /^type:[[:space:]]*(Feature|Bug|Decision|Research|Investigation|Spike)[[:space:]]*$/) has_type = 1 } # Only the explicit task-card schema authorizes automatic relocation. # Generic body prose after a licensed header is never moved on key: alone. @@ -292,8 +298,12 @@ insert_header() { # Keep complete Markdown metadata ahead of the license in repair mode too. local metadata_lines="" - if [[ "$f" == *.md && "$first_line" == "---" ]]; then - metadata_lines=$(awk 'NR > 1 && $0 == "---" { print NR; exit }' "$f") + if [[ "$f" == *.md ]]; then + local metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" + if [[ "$metadata_start" -eq 1 && "$metadata_end" -gt 0 ]]; then + metadata_lines="$metadata_end" + fi fi # Logic to insert header AFTER metadata/shebang/xml declaration if present diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index cd45a40f..79745b96 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -6,7 +6,7 @@ root=$(git rev-parse --show-toplevel) cd "$root" checker=${1:-scripts/ensure_spdx.sh} case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -198,4 +198,51 @@ bash "$checker" --check "$work/equals-comment.md" test "$(grep -c 'SPDX-License-Identifier' "$work/equals-comment.md")" = 1 grep -qx '# Task' "$work/equals-comment.md" fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == indented_keys ]]; then +cat > "$work/indented-keys.md" <<'DOC' + + + +--- +Example: a task-card code sample follows. + + id: S01 + type: Feature +--- +DOC +cp "$work/indented-keys.md" "$work/indented-keys.original" +bash "$checker" --check "$work/indented-keys.md" +bash "$checker" "$work/indented-keys.md" +cmp "$work/indented-keys.original" "$work/indented-keys.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == indented_comments ]]; then +cat > "$work/indented-comments.md" <<'DOC' +--- +id: S01 +--- + + + + +Body. +DOC +tail -n +4 "$work/indented-comments.md" > "$work/indented-comments.expected" +if bash "$checker" "$work/indented-comments.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/indented-comments.md" +tail -n +6 "$work/indented-comments.md" > "$work/indented-comments.actual" +cmp "$work/indented-comments.expected" "$work/indented-comments.actual" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == quoted_keys ]]; then +for key in '"title"' "'title'"; do +printf -- '---\n%s: Example\n---\n# Body\n' "$key" > "$work/quoted-keys.md" +head -n 3 "$work/quoted-keys.md" > "$work/quoted-keys.metadata" +if bash "$checker" "$work/quoted-keys.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/quoted-keys.md" +cp "$work/quoted-keys.md" "$work/quoted-keys.once" +bash "$checker" "$work/quoted-keys.md" +cmp "$work/quoted-keys.once" "$work/quoted-keys.md" +head -n 3 "$work/quoted-keys.md" > "$work/quoted-keys.actual" +cmp "$work/quoted-keys.metadata" "$work/quoted-keys.actual" +done +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' From 89d4d4c13eb4b180905af91039544a8949106ed7 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 09:30:04 -0700 Subject: [PATCH 14/27] docs: state top-level task metadata relocation markers --- docs/DOCUMENTATION_STANDARDS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index 3d7c8a97..baf627b2 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,7 +163,7 @@ when one exists. ## Writing and structure -Markdown frontmatter remains first. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with a nonempty `id` and a recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. +Markdown frontmatter remains first. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. From 42a717301bd7e315f2aed1149112c5a9d1856bf1 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 09:53:15 -0700 Subject: [PATCH 15/27] fix(tooling): preserve spaced map keys and reject duplicate license headers --- scripts/ensure_spdx.sh | 26 ++++++++++++++++------ scripts/tests/spdx_frontmatter_test.sh | 30 +++++++++++++++++++++++++- 2 files changed, 49 insertions(+), 7 deletions(-) diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index b2c3bcee..da553382 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -115,9 +115,11 @@ markdown_metadata_bounds() { awk -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' function map_key(line, quote) { quote = sprintf("%c", 39) - return line ~ /^[[:space:]]*[[:alpha:]_][[:alnum:]_.-]*:([[:space:]]|$)/ || - line ~ /^[[:space:]]*".*"[[:space:]]*:([[:space:]]|$)/ || - line ~ ("^[[:space:]]*" quote ".*" quote "[[:space:]]*:([[:space:]]|$)") + trimmed = line + sub(/^[[:space:]]*/, "", trimmed) + if (substr(trimmed, 1, 1) == "\"") return trimmed ~ /^".*"[[:space:]]*:([[:space:]]|$)/ + if (substr(trimmed, 1, 1) == quote) return trimmed ~ ("^" quote ".*" quote "[[:space:]]*:([[:space:]]|$)") + return trimmed ~ /.+:([[:space:]]|$)/ } { lines[NR] = $0 } END { @@ -144,8 +146,8 @@ markdown_metadata_bounds() { has_type = 0 for (i = start + 1; i <= NR; i++) { if (lines[i] == "---") { finish = i; break } - if (lines[i] ~ /^id:[[:space:]]*[^[:space:]]/) has_id = 1 - if (lines[i] ~ /^type:[[:space:]]*(Feature|Bug|Decision|Research|Investigation|Spike)[[:space:]]*$/) has_type = 1 + if (lines[i] ~ /^id[[:space:]]*:[[:space:]]*[^[:space:]]/) has_id = 1 + if (lines[i] ~ /^type[[:space:]]*:[[:space:]]*(Feature|Bug|Decision|Research|Investigation|Spike)[[:space:]]*$/) has_type = 1 } # Only the explicit task-card schema authorizes automatic relocation. # Generic body prose after a licensed header is never moved on key: alone. @@ -198,7 +200,19 @@ check_valid_header() { # Compare line by line if [[ "${file_lines[i]:-}" == "${expected_lines[0]}" && "${file_lines[i+1]:-}" == "${expected_lines[1]}" ]]; then - return 0 # Exact header found + if [[ "$f" == *.md ]]; then + # Reject a conflicting declaration in the same bounded header region. + awk -v first="$((i + 3))" -v last="$((i + 15))" \ + -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + NR < first { next } + NR > last { exit } + /^[[:space:]]*$/ { next } + $0 ~ license_re || $0 ~ copyright_re { duplicate = 1; exit } + { exit } + END { exit duplicate ? 1 : 0 } + ' "$f" || return 1 + fi + return 0 # Exact unique header found fi return 1 } diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 79745b96..87d26446 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -6,7 +6,7 @@ root=$(git rev-parse --show-toplevel) cd "$root" checker=${1:-scripts/ensure_spdx.sh} case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -245,4 +245,32 @@ head -n 3 "$work/quoted-keys.md" > "$work/quoted-keys.actual" cmp "$work/quoted-keys.metadata" "$work/quoted-keys.actual" done fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == spaced_keys ]]; then +cat > "$work/spaced-keys.md" <<'DOC' +--- +title : Example +--- +# Body +DOC +head -n 3 "$work/spaced-keys.md" > "$work/spaced-keys.expected" +if bash "$checker" "$work/spaced-keys.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/spaced-keys.md" +head -n 3 "$work/spaced-keys.md" > "$work/spaced-keys.actual" +cmp "$work/spaced-keys.expected" "$work/spaced-keys.actual" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == duplicate_headers ]]; then +cat > "$work/duplicate-header.md" <<'DOC' +--- +id: S01 +--- + + + +# Body +DOC +if bash "$checker" --check "$work/duplicate-header.md"; then exit 1; else test "$?" = 1; fi +if bash "$checker" "$work/duplicate-header.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/duplicate-header.md" +test "$(grep -c SPDX-License-Identifier "$work/duplicate-header.md")" = 1 +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' From 06aca976a0237daaab5072f986655f4cec8e576a Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 10:29:28 -0700 Subject: [PATCH 16/27] fix(tooling): preserve complete metadata framing and refuse unclosed headers --- docs/DOCUMENTATION_STANDARDS.md | 2 +- scripts/ensure_spdx.sh | 31 ++++++++++++++++++++++++-- scripts/tests/spdx_frontmatter_test.sh | 29 +++++++++++++++++++++++- 3 files changed, 58 insertions(+), 4 deletions(-) diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index baf627b2..f38aee51 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,7 +163,7 @@ when one exists. ## Writing and structure -Markdown frontmatter remains first. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. +Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed metadata or license-header attempts refuse repair without mutation. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index da553382..725dabd8 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -135,10 +135,16 @@ markdown_metadata_bounds() { } } if (!start) { print 0, 0; exit } - # A thematic break is body content. Task frontmatter begins with a map. + # A complete line-one delimiter block is metadata; YAML grammar belongs + # to its consumer. Legacy header-first relocation still needs task keys. + if (start == 1) { + for (i = 2; i <= NR; i++) { + if (lines[i] == "---") { print 1, i; exit } + } + } first = start + 1 while (first <= NR && (lines[first] ~ /^[[:space:]]*$/ || lines[first] ~ /^[[:space:]]*#/)) first++ - if (first > NR || !map_key(lines[first])) { + if (first > NR || (!map_key(lines[first]) && lines[first] !~ /^\?[[:space:]]/)) { print 0, 0; exit } finish = 0 @@ -342,6 +348,22 @@ insert_header() { rm "$temp_file" } + +markdown_has_unclosed_header_attempt() { + local file="$1" metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$file")" + awk -v first="$((metadata_end + 1))" -v last="$((metadata_end + 15))" \ + -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + NR < first { next } + NR > last { exit } + /^[[:space:]]*$/ { next } + /^/ { malformed = 1; exit } + $0 ~ license_re || $0 ~ copyright_re { next } + { exit } + END { exit malformed ? 0 : 1 } + ' "$file" +} + process_file() { local f="$1" if should_skip "$f"; then return; fi @@ -361,6 +383,11 @@ process_file() { FAILED_COUNT=$((FAILED_COUNT + 1)) return fi + if markdown_has_unclosed_header_attempt "$f"; then + echo "[FAIL] Cannot process Markdown with an unclosed license header: $f" + FAILED_COUNT=$((FAILED_COUNT + 1)) + return + fi fi if check_valid_header "$f" "$expected_header_block"; then diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 87d26446..9e5be7aa 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -6,7 +6,7 @@ root=$(git rev-parse --show-toplevel) cd "$root" checker=${1:-scripts/ensure_spdx.sh} case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -273,4 +273,31 @@ if bash "$checker" "$work/duplicate-header.md"; then exit 1; else test "$?" = 1; bash "$checker" --check "$work/duplicate-header.md" test "$(grep -c SPDX-License-Identifier "$work/duplicate-header.md")" = 1 fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == explicit_keys ]]; then +cat > "$work/explicit-keys.md" <<'DOC' +--- +? title +: Example +--- +# Body +DOC +head -n 4 "$work/explicit-keys.md" > "$work/explicit-keys.expected" +if bash "$checker" "$work/explicit-keys.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/explicit-keys.md" +head -n 4 "$work/explicit-keys.md" > "$work/explicit-keys.actual" +cmp "$work/explicit-keys.expected" "$work/explicit-keys.actual" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == unclosed_license ]]; then +cat > "$work/unclosed-license.md" <<'DOC' +--- +id: S01 +--- +[[:space:]]*$' +MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*$' MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*$' markdown_metadata_bounds() { @@ -121,7 +121,7 @@ markdown_metadata_bounds() { if (substr(trimmed, 1, 1) == quote) return trimmed ~ ("^" quote ".*" quote "[[:space:]]*:([[:space:]]|$)") return trimmed ~ /.+:([[:space:]]|$)/ } - { lines[NR] = $0 } + { parsed = $0; sub(/\r$/, "", parsed); lines[NR] = parsed } END { start = 0 if (lines[1] == "---") start = 1 @@ -153,7 +153,16 @@ markdown_metadata_bounds() { for (i = start + 1; i <= NR; i++) { if (lines[i] == "---") { finish = i; break } if (lines[i] ~ /^id[[:space:]]*:[[:space:]]*[^[:space:]]/) has_id = 1 - if (lines[i] ~ /^type[[:space:]]*:[[:space:]]*(Feature|Bug|Decision|Research|Investigation|Spike)[[:space:]]*$/) has_type = 1 + if (lines[i] ~ /^type[[:space:]]*:/) { + value = lines[i] + sub(/^type[[:space:]]*:[[:space:]]*/, "", value) + sub(/[[:space:]]*$/, "", value) + quote = substr(value, 1, 1) + if ((quote == "\"" || quote == sprintf("%c", 39)) && substr(value, length(value), 1) == quote) { + value = substr(value, 2, length(value) - 2) + } + if (value ~ /^(Feature|Bug|Decision|Research|Investigation|Spike)$/) has_type = 1 + } } # Only the explicit task-card schema authorizes automatic relocation. # Generic body prose after a licensed header is never moved on key: alone. @@ -170,6 +179,7 @@ check_valid_header() { local file_lines=() # Read file line by line, handling newlines properly while IFS= read -r line; do + if [[ "$f" == *.md ]]; then line="${line%$'\r'}"; fi file_lines+=("$line") done < "$f" @@ -315,6 +325,11 @@ insert_header() { local first_line first_line=$(head -n 1 "$f" || true) + local header_cr="" + if [[ "$f" == *.md && "$first_line" == *$'\r' ]]; then + header_cr=$'\r' + header="${header//$'\n'/$'\r\n'}" + fi # Keep complete Markdown metadata ahead of the license in repair mode too. local metadata_lines="" @@ -329,18 +344,18 @@ insert_header() { # Logic to insert header AFTER metadata/shebang/xml declaration if present if [[ -n "$metadata_lines" ]]; then head -n "$metadata_lines" "$f" > "$temp_file" - echo "$header" >> "$temp_file" + printf '%s%s\n' "$header" "$header_cr" >> "$temp_file" tail -n "+$((metadata_lines + 1))" "$f" >> "$temp_file" elif [[ "$first_line" =~ ^#! ]]; then echo "$first_line" > "$temp_file" - echo "$header" >> "$temp_file" + printf '%s%s\n' "$header" "$header_cr" >> "$temp_file" tail -n +2 "$f" >> "$temp_file" elif [[ "$first_line" =~ ^\<\?xml ]]; then echo "$first_line" > "$temp_file" - echo "$header" >> "$temp_file" + printf '%s%s\n' "$header" "$header_cr" >> "$temp_file" tail -n +2 "$f" >> "$temp_file" else - echo "$header" > "$temp_file" + printf '%s%s\n' "$header" "$header_cr" > "$temp_file" cat "$f" >> "$temp_file" fi diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 9e5be7aa..77800e87 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -6,7 +6,7 @@ root=$(git rev-parse --show-toplevel) cd "$root" checker=${1:-scripts/ensure_spdx.sh} case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -300,4 +300,52 @@ if bash "$checker" "$work/unclosed-license.md"; then exit 1; else test "$?" = 1; cmp "$work/unclosed-license.original" "$work/unclosed-license.md" if bash "$checker" --check "$work/unclosed-license.md"; then exit 1; else test "$?" = 1; fi fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == quoted_type ]]; then +cat > "$work/quoted-type.md" <<'DOC' + + +--- +id: S01 +type: "Feature" +--- +# Body +DOC +if bash "$checker" --check "$work/quoted-type.md"; then exit 1; else test "$?" = 1; fi +if bash "$checker" "$work/quoted-type.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/quoted-type.md" +test "$(head -n 1 "$work/quoted-type.md")" = '---' +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == crlf ]]; then +printf -- '---\r\nid: S01\r\n---\r\n# Body\r\n' > "$work/crlf.md" +head -n 3 "$work/crlf.md" > "$work/crlf.metadata" +tail -n +4 "$work/crlf.md" > "$work/crlf.body" +if bash "$checker" "$work/crlf.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/crlf.md" +head -n 3 "$work/crlf.md" > "$work/crlf.actual-metadata" +tail -n +6 "$work/crlf.md" > "$work/crlf.actual-body" +cmp "$work/crlf.metadata" "$work/crlf.actual-metadata" +cmp "$work/crlf.body" "$work/crlf.actual-body" +python3 - "$work/crlf.md" <<'CHECK' +from pathlib import Path +import sys +raw=Path(sys.argv[1]).read_bytes() +assert b"\n" not in raw.replace(b"\r\n", b"") +CHECK +cp "$work/crlf.md" "$work/crlf.once" +bash "$checker" "$work/crlf.md" +cmp "$work/crlf.once" "$work/crlf.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == delimiterless ]]; then +cat > "$work/delimiterless.md" <<'DOC' +--- +id: S01 +--- + + +# Body +DOC +if bash "$checker" "$work/delimiterless.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/delimiterless.md" +test "$(grep -c SPDX-License-Identifier "$work/delimiterless.md")" = 1 +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' diff --git a/tasks/K01.md b/tasks/K01.md index 2e6d7632..04c2e000 100644 --- a/tasks/K01.md +++ b/tasks/K01.md @@ -27,7 +27,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Prove the Echo and Keep content identity bridge. Current Echo has no physical-content port or Keep adapter. +Prove the Echo and Keep content identity bridge. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. ## 3. Proposed Solution diff --git a/tasks/K02.md b/tasks/K02.md index a698083e..21bfe9b0 100644 --- a/tasks/K02.md +++ b/tasks/K02.md @@ -30,7 +30,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Add the Echo physical-content port and existing CAS adapters. Current Echo has no physical-content port or Keep adapter. +Add the Echo physical-content port and existing CAS adapters. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. ## 3. Proposed Solution diff --git a/tasks/K03.md b/tasks/K03.md index 1ae786c1..a8fb1bcc 100644 --- a/tasks/K03.md +++ b/tasks/K03.md @@ -27,7 +27,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Add an experimental Keep ReferenceStore adapter. Current Echo has no physical-content port or Keep adapter. +Add an experimental Keep ReferenceStore adapter. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. ## 3. Proposed Solution diff --git a/tasks/K04.md b/tasks/K04.md index c9bcb1c9..67a315bd 100644 --- a/tasks/K04.md +++ b/tasks/K04.md @@ -28,7 +28,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Add a pinned-generation durable Keep read adapter. Current Echo has no physical-content port or Keep adapter. +Add a pinned-generation durable Keep read adapter. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. ## 3. Proposed Solution diff --git a/tasks/K05.md b/tasks/K05.md index e2e46495..13769d6c 100644 --- a/tasks/K05.md +++ b/tasks/K05.md @@ -29,7 +29,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Prove durable Echo and Keep publication reconciliation. Current Echo has no physical-content port or Keep adapter. +Prove durable Echo and Keep publication reconciliation. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. ## 3. Proposed Solution diff --git a/tasks/K06.md b/tasks/K06.md index a54ac852..c92252e8 100644 --- a/tasks/K06.md +++ b/tasks/K06.md @@ -28,7 +28,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Prepare the migration and production adoption decision. Current Echo has no physical-content port or Keep adapter. +Prepare the migration and production adoption decision. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. ## 3. Proposed Solution From e3af939d9213c40cde3890938219fb6fe0f58fa7 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 13:06:20 -0700 Subject: [PATCH 19/27] fix(docs): recognize nonempty task IDs and commented types --- scripts/ensure_spdx.sh | 13 ++++++++++- scripts/tests/spdx_frontmatter_test.sh | 32 +++++++++++++++++++++++++- 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index 2a44b625..3977f159 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -152,10 +152,21 @@ markdown_metadata_bounds() { has_type = 0 for (i = start + 1; i <= NR; i++) { if (lines[i] == "---") { finish = i; break } - if (lines[i] ~ /^id[[:space:]]*:[[:space:]]*[^[:space:]]/) has_id = 1 + if (lines[i] ~ /^id[[:space:]]*:/) { + id_value = lines[i] + sub(/^id[[:space:]]*:[[:space:]]*/, "", id_value) + sub(/[[:space:]]+#.*$/, "", id_value) + sub(/[[:space:]]*$/, "", id_value) + id_quote = substr(id_value, 1, 1) + if ((id_quote == "\"" || id_quote == sprintf("%c", 39)) && substr(id_value, length(id_value), 1) == id_quote) { + id_value = substr(id_value, 2, length(id_value) - 2) + } + if (id_value != "" && id_value != "null" && id_value != "~") has_id = 1 + } if (lines[i] ~ /^type[[:space:]]*:/) { value = lines[i] sub(/^type[[:space:]]*:[[:space:]]*/, "", value) + sub(/[[:space:]]+#.*$/, "", value) sub(/[[:space:]]*$/, "", value) quote = substr(value, 1, 1) if ((quote == "\"" || quote == sprintf("%c", 39)) && substr(value, length(value), 1) == quote) { diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 77800e87..c868c705 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -6,7 +6,7 @@ root=$(git rev-parse --show-toplevel) cd "$root" checker=${1:-scripts/ensure_spdx.sh} case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -348,4 +348,34 @@ if bash "$checker" "$work/delimiterless.md"; then exit 1; else test "$?" = 1; fi bash "$checker" --check "$work/delimiterless.md" test "$(grep -c SPDX-License-Identifier "$work/delimiterless.md")" = 1 fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == empty_id ]]; then +cat > "$work/empty-id.md" <<'DOC' + + +--- +id: "" +type: Feature +--- +# Body +DOC +cp "$work/empty-id.md" "$work/empty-id.original" +bash "$checker" --check "$work/empty-id.md" +bash "$checker" "$work/empty-id.md" +cmp "$work/empty-id.original" "$work/empty-id.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == type_comment ]]; then +cat > "$work/type-comment.md" <<'DOC' + + +--- +id: S01 +type: Feature # task category +--- +# Body +DOC +if bash "$checker" --check "$work/type-comment.md"; then exit 1; else test "$?" = 1; fi +if bash "$checker" "$work/type-comment.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/type-comment.md" +test "$(head -n 1 "$work/type-comment.md")" = '---' +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' From c46456565d6cfbd92fce041fe0c7b055147e1fee Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 13:13:12 -0700 Subject: [PATCH 20/27] docs(planning): record the merged identity prerequisite --- ROADMAP.md | 6 +++--- tasks/K01.md | 4 ++-- tasks/K03.md | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 7f94ce0f..05f63efe 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -68,7 +68,7 @@ Echo selects Rust 1.96.0 for its toolchain and warp-core. The workspace default Keep main exports `DurableStore` and fenced snapshots. Its crate documentation still states that production durable ingestion, garbage collection, and compaction are unimplemented. Thus durable reads and durable publication have different readiness. -- [ ] [K01: Prove the Echo and Keep content identity bridge](tasks/K01.md) — [issue #759](https://github.com/flyingrobots/echo/issues/759) +- [x] [K01: Prove the Echo and Keep content identity bridge](tasks/K01.md) — [issue #759](https://github.com/flyingrobots/echo/issues/759) - [x] [K02: Add the Echo physical-content port and existing CAS adapters](tasks/K02.md) — [issue #760](https://github.com/flyingrobots/echo/issues/760) - [ ] [K03: Add an experimental Keep ReferenceStore adapter](tasks/K03.md) — [issue #761](https://github.com/flyingrobots/echo/issues/761) - [ ] [K04: Add a pinned-generation durable Keep read adapter](tasks/K04.md) @@ -83,7 +83,7 @@ Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. -The initial proposed graph has ten task vertices and five internal edges. S01, S02, S03, S04, and K02 are complete. The remaining graph has five task vertices and four internal edges; its first layer is `{K01}`. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. +The initial proposed graph has ten task vertices and five internal edges. S01, S02, S03, S04, K01, and K02 are complete. The remaining graph has four task vertices and three internal edges; its first layer is `{K03}`. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. @@ -99,7 +99,7 @@ The independent critique confirms the S01 contract repair and experimental K01 ## Verification and merge rules -Use the existing `echo-read-runtime` worker and stable `/lease-target` cache. Use `/Users/Shared/git-locks/workstation.git` with `host/heavy-work` and `host/docker/echo-read-runtime/`. The monitor also inspects `echo-provider-builder`; reserve its key during validation. +Use the existing `echo-read-runtime` worker and stable `/lease-target` cache. Use `/Users/Shared/git-locks/workstation.git` with the concrete worker key `host/docker/echo-read-runtime/`. The monitor also inspects `echo-provider-builder`; reserve its key during validation. Keep the 20 GiB build, 4 GiB data, and 128 MiB log limits. Keep the 50 GiB host and VM free-space floors. Use copied source, four CPUs, 6 GiB RAM, bounded logs, process deadlines, and the fail-closed guard. Do not use host test fallbacks. diff --git a/tasks/K01.md b/tasks/K01.md index 04c2e000..31341562 100644 --- a/tasks/K01.md +++ b/tasks/K01.md @@ -39,7 +39,7 @@ Use an isolated experimental adapter package with an explicit Rust 1.96 requirem ## 4. Prerequisites -- [ ] Recheck exact Echo and Keep identity source and supported toolchains. +- [x] Recheck exact Echo and Keep identity source and supported toolchains. Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. @@ -55,7 +55,7 @@ A golden-vector suite proves same-source and reconstructed-source agreement. A w ## 7. Definition of Done -The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. +Completed in [PR #768](https://github.com/flyingrobots/echo/pull/768), integrated at `bb20c57345374f476fa938789294a0890341eadd`. Six identity witnesses, relevant guarded Docker and hosted checks, and current-head Code Lawyer and agy review passed. The bridge remains isolated and supplies no storage or durability proposition. ## 8. Test Plan diff --git a/tasks/K03.md b/tasks/K03.md index a8fb1bcc..eadf7a8d 100644 --- a/tasks/K03.md +++ b/tasks/K03.md @@ -35,8 +35,8 @@ Pin the inspected Keep revision. Add an optional Echo adapter over Keep Referenc ## 4. Prerequisites -- [ ] Complete [K01](K01.md). Its contract supplies required behavior. -- [ ] Complete [K02](K02.md). Its contract supplies required behavior. +- [x] Complete [K01](K01.md). Its contract supplies required behavior. +- [x] Complete [K02](K02.md). Its contract supplies required behavior. Recheck external capability prerequisites before execution. Record accepted edges on #722. GitHub records #759 and #760 as blockers of #761. Recheck the accepted edges before execution. From 71990c28f4929934a3aae7fc4f3a762c6093c7b2 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 13:16:21 -0700 Subject: [PATCH 21/27] fix(docs): refuse unclosed sequence metadata without mutation --- ROADMAP.md | 4 ++-- scripts/ensure_spdx.sh | 2 +- scripts/tests/spdx_frontmatter_test.sh | 16 +++++++++++++++- 3 files changed, 18 insertions(+), 4 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 05f63efe..828b4f8f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -50,7 +50,7 @@ Code paths are relative to this repository, except the explicitly named hello-ec - [x] [S03: Report the typed Action obstruction when the operation runner cannot commit](tasks/S03.md) — [issue #756](https://github.com/flyingrobots/echo/issues/756) - [x] [S04: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery](tasks/S04.md) — [issue #757](https://github.com/flyingrobots/echo/issues/757) -Execute S01, S02, S03, then S04. Each issue produces one independently mergeable PR. This order puts smaller contract and runner repairs before recovery work. +The original order was S01, S02, S03, then S04. Each repair landed in one independently mergeable PR. This order put smaller contract and runner repairs before recovery work. The feedback-repair graph has four vertices and zero required edges. All four tasks form one antichain at the audited revision. The chosen sequence serializes the shared worker; it does not imply a code prerequisite. @@ -83,7 +83,7 @@ Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. -The initial proposed graph has ten task vertices and five internal edges. S01, S02, S03, S04, K01, and K02 are complete. The remaining graph has four task vertices and three internal edges; its first layer is `{K03}`. Its layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. +The initial proposed graph has ten task vertices and five internal edges. S01, S02, S03, S04, K01, and K02 are complete. The remaining graph has four task vertices and three internal edges; its first layer is `{K03}`. The initial graph layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index 3977f159..c2008198 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -144,7 +144,7 @@ markdown_metadata_bounds() { } first = start + 1 while (first <= NR && (lines[first] ~ /^[[:space:]]*$/ || lines[first] ~ /^[[:space:]]*#/)) first++ - if (first > NR || (!map_key(lines[first]) && lines[first] !~ /^\?[[:space:]]/)) { + if (first > NR || (!map_key(lines[first]) && lines[first] !~ /^\?[[:space:]]/ && lines[first] !~ /^[[:space:]]*-([[:space:]]|$)/)) { print 0, 0; exit } finish = 0 diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index c868c705..920420eb 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -5,8 +5,11 @@ set -euo pipefail root=$(git rev-parse --show-toplevel) cd "$root" checker=${1:-scripts/ensure_spdx.sh} +# Every selected regression first runs the required valid/missing/wrong/header +# baseline. Selectors narrow the added regression, not this preflight. A RED +# assertion is evidence only after the baseline passes. case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -378,4 +381,15 @@ if bash "$checker" "$work/type-comment.md"; then exit 1; else test "$?" = 1; fi bash "$checker" --check "$work/type-comment.md" test "$(head -n 1 "$work/type-comment.md")" = '---' fi + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == sequence_root ]]; then +for sequence_item in '- item' ' - item' '-'; do + printf '%s\n' '---' "$sequence_item" > "$work/sequence-root.md" + cp "$work/sequence-root.md" "$work/sequence-root.original" + if bash "$checker" --check "$work/sequence-root.md"; then exit 1; fi + if bash "$checker" "$work/sequence-root.md"; then exit 1; else test "$?" = 1; fi + cmp "$work/sequence-root.original" "$work/sequence-root.md" +done +fi + printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' From e40243f3d235c90a443079efd75a8171216539c9 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 13:44:32 -0700 Subject: [PATCH 22/27] docs(planning): close experimental scope and retain production gates --- ROADMAP.md | 8 ++++---- docs/DOCUMENTATION_STANDARDS.md | 2 +- tasks/K01.md | 2 +- tasks/K02.md | 2 +- tasks/K03.md | 8 ++++---- tasks/K04.md | 6 +++--- tasks/K05.md | 2 +- tasks/K06.md | 2 +- 8 files changed, 16 insertions(+), 16 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 828b4f8f..d6477af1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -70,12 +70,12 @@ Keep main exports `DurableStore` and fenced snapshots. Its crate documentation s - [x] [K01: Prove the Echo and Keep content identity bridge](tasks/K01.md) — [issue #759](https://github.com/flyingrobots/echo/issues/759) - [x] [K02: Add the Echo physical-content port and existing CAS adapters](tasks/K02.md) — [issue #760](https://github.com/flyingrobots/echo/issues/760) -- [ ] [K03: Add an experimental Keep ReferenceStore adapter](tasks/K03.md) — [issue #761](https://github.com/flyingrobots/echo/issues/761) +- [x] [K03: Add an experimental Keep ReferenceStore adapter](tasks/K03.md) — [issue #761](https://github.com/flyingrobots/echo/issues/761) - [ ] [K04: Add a pinned-generation durable Keep read adapter](tasks/K04.md) - [ ] [K05: Prove durable Echo and Keep publication reconciliation](tasks/K05.md) - [ ] [K06: Prepare the migration and production adoption decision](tasks/K06.md) -Execute K01 and K02 as independent slices, then K03. Stop this run after the experimental adapter passes its conformance and review gates. +K01 and K02 landed as independent slices, followed by K03. The experimental adapter passed its conformance and review gates in [PR #770](https://github.com/flyingrobots/echo/pull/770), integrated at `fe8789263a26fbcb7c7554c2b48f9c33b812c7eb`. This completes the authorized implementation scope. K04–K06 are conditional follow-on work. K04 requires guarded storage that actually passes Keep's Linux ext4 admission. K05 is blocked on external Keep durable ingestion, operation lookup, and non-expiring retention anchors. K06 also requires crash, migration, rollback, and adoption evidence. Completing K03 does not unblock these requirements. @@ -83,7 +83,7 @@ Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. -The initial proposed graph has ten task vertices and five internal edges. S01, S02, S03, S04, K01, and K02 are complete. The remaining graph has four task vertices and three internal edges; its first layer is `{K03}`. The initial graph layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. +The initial proposed graph has ten task vertices and five internal edges. S01, S02, S03, S04, K01, K02, and K03 are complete. The remaining conditional graph has three task vertices and two internal edges; its first structural layer is `{K04}`. No conditional task is ready for this run: external gates and the separate adoption decision remain unmet. The initial graph layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. @@ -93,7 +93,7 @@ S03 reports bounded outcome categories. It distinguishes a missing outcome from S04 indexes exact obligations before replay. It sorts verification references within each worldline and keeps retained protocol order authoritative. Its two-sweep plan supports stale and cross-worldline bases. Replay counters bound patch applications; they do not prove linear elapsed time. -K01 verifies two distinct identity laws on the same source and reconstructed bytes. Equal source bytes do not make Echo and Keep digest values equal. K02 preserves existing CAS defaults and package compatibility. +K01 verifies two distinct identity laws on the same source and reconstructed bytes. Equal source bytes do not make Echo and Keep digest values equal. K02 preserves existing CAS defaults and package compatibility. K03 keeps backend error causes private as well as binding fields: public formatting, downcast and source chains must not expose Keep coordinates. A late independent finding demonstrated this leak; the corrected adapter passed its regression and a fresh review. The independent critique confirms the S01 contract repair and experimental K01–K03 boundary. Its Reader source is receipt `7e05db3c-49d0-45df-b81b-2f1a6a25a6f7`. Reconciliation details remain on PR #758; this plan contains the resulting requirements. diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index f38aee51..1ec6a6d0 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,7 +163,7 @@ when one exists. ## Writing and structure -Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed metadata or license-header attempts refuse repair without mutation. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. +Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key or sequence hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. diff --git a/tasks/K01.md b/tasks/K01.md index 31341562..6b98b7a1 100644 --- a/tasks/K01.md +++ b/tasks/K01.md @@ -27,7 +27,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Prove the Echo and Keep content identity bridge. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. +Prove the Echo and Keep content identity bridge. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. ## 3. Proposed Solution diff --git a/tasks/K02.md b/tasks/K02.md index 21bfe9b0..1cb4331d 100644 --- a/tasks/K02.md +++ b/tasks/K02.md @@ -30,7 +30,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Add the Echo physical-content port and existing CAS adapters. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. +Add the Echo physical-content port and existing CAS adapters. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. ## 3. Proposed Solution diff --git a/tasks/K03.md b/tasks/K03.md index eadf7a8d..db5df376 100644 --- a/tasks/K03.md +++ b/tasks/K03.md @@ -27,11 +27,11 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Add an experimental Keep ReferenceStore adapter. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. +Add an experimental Keep ReferenceStore adapter. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. ## 3. Proposed Solution -Pin the inspected Keep revision. Add an optional Echo adapter over Keep ReferenceStore. Reuse the port conformance suite. Keep the adapter disabled by default and label its receipts non-durable. +Pin the inspected Keep revision. Add an optional Echo adapter over Keep ReferenceStore. Reuse the port conformance suite. Keep the adapter disabled by default and label its receipts non-durable. Keep concrete backend causes and coordinates private on error paths, including formatting, downcast and source chains. ## 4. Prerequisites @@ -52,13 +52,13 @@ Both existing CAS and Keep pass one backend-neutral suite. The adapter changes n ## 7. Definition of Done -The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. +Completed in [PR #770](https://github.com/flyingrobots/echo/pull/770), integrated at `fe8789263a26fbcb7c7554c2b48f9c33b812c7eb`. Thirteen feature-enabled witnesses, six default identity witnesses, the 36-test CAS suite, relevant guarded Docker and hosted checks, and current-head Code Lawyer and agy review passed. The adapter remains disabled by default and volatile. Its error boundary keeps concrete Keep causes private. ## 8. Test Plan Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. -Edges: Byte equality, identity substitution, missing chunks, corrupt layouts, deterministic layout selection, capacity limits, and source interruption. +Edges: Byte equality, identity substitution, missing chunks, corrupt layouts, deterministic layout selection, capacity limits, source interruption, and public error-coordinate isolation. Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. diff --git a/tasks/K04.md b/tasks/K04.md index 67a315bd..e1ca7d3f 100644 --- a/tasks/K04.md +++ b/tasks/K04.md @@ -28,7 +28,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Add a pinned-generation durable Keep read adapter. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. +Add a pinned-generation durable Keep read adapter. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. ## 3. Proposed Solution @@ -38,11 +38,11 @@ First prove Keep admission on the exact backing filesystem. The durable profile Run required tests in Docker. Do not use host test fallbacks. Unsupported-platform checks require a permitted Docker or cross-target route that preserves the execution policy. - Add an optional adapter for an admitted immutable generation. Retain its evidence during the read. Reconstruct into bounded private staging and validate the full receipt. +Add an optional adapter for an admitted immutable generation. Retain its evidence during the read. Reconstruct into bounded private staging and validate the full receipt. ## 4. Prerequisites -- [ ] Complete [K03](K03.md). Its contract supplies required behavior. +- [x] Complete [K03](K03.md). Its contract supplies required behavior. Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. diff --git a/tasks/K05.md b/tasks/K05.md index 13769d6c..29fb2563 100644 --- a/tasks/K05.md +++ b/tasks/K05.md @@ -29,7 +29,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Prove durable Echo and Keep publication reconciliation. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. +Prove durable Echo and Keep publication reconciliation. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. ## 3. Proposed Solution diff --git a/tasks/K06.md b/tasks/K06.md index c92252e8..d9f95ac5 100644 --- a/tasks/K06.md +++ b/tasks/K06.md @@ -28,7 +28,7 @@ Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `316 ## 2. Problem Description -Prepare the migration and production adoption decision. Echo has the merged physical-content port and MemoryTier/DiskTier adapters. The experimental Keep backend remains to be implemented. +Prepare the migration and production adoption decision. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. ## 3. Proposed Solution From 4d66baded8f64d54a74e4bfb3523095fc8f76f0f Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 13:52:28 -0700 Subject: [PATCH 23/27] fix(docs): preserve comment-only IDs and unclosed flow metadata --- docs/DOCUMENTATION_STANDARDS.md | 2 +- scripts/ensure_spdx.sh | 8 ++++++- scripts/tests/spdx_frontmatter_test.sh | 29 +++++++++++++++++++++++++- 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index 1ec6a6d0..b52b9f33 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,7 +163,7 @@ when one exists. ## Writing and structure -Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key or sequence hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. +Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key, block-sequence or flow-container hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index c2008198..a0e52641 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -121,6 +121,11 @@ markdown_metadata_bounds() { if (substr(trimmed, 1, 1) == quote) return trimmed ~ ("^" quote ".*" quote "[[:space:]]*:([[:space:]]|$)") return trimmed ~ /.+:([[:space:]]|$)/ } + function metadata_hint(line, first_char) { + sub(/^[[:space:]]*/, "", line) + first_char = substr(line, 1, 1) + return map_key(line) || line ~ /^\?[[:space:]]/ || line ~ /^-([[:space:]]|$)/ || first_char == "[" || first_char == "{" + } { parsed = $0; sub(/\r$/, "", parsed); lines[NR] = parsed } END { start = 0 @@ -144,7 +149,7 @@ markdown_metadata_bounds() { } first = start + 1 while (first <= NR && (lines[first] ~ /^[[:space:]]*$/ || lines[first] ~ /^[[:space:]]*#/)) first++ - if (first > NR || (!map_key(lines[first]) && lines[first] !~ /^\?[[:space:]]/ && lines[first] !~ /^[[:space:]]*-([[:space:]]|$)/)) { + if (first > NR || !metadata_hint(lines[first])) { print 0, 0; exit } finish = 0 @@ -155,6 +160,7 @@ markdown_metadata_bounds() { if (lines[i] ~ /^id[[:space:]]*:/) { id_value = lines[i] sub(/^id[[:space:]]*:[[:space:]]*/, "", id_value) + if (id_value ~ /^#/) id_value = "" sub(/[[:space:]]+#.*$/, "", id_value) sub(/[[:space:]]*$/, "", id_value) id_quote = substr(id_value, 1, 1) diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 920420eb..27aaebe4 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -9,7 +9,7 @@ checker=${1:-scripts/ensure_spdx.sh} # baseline. Selectors narrow the added regression, not this preflight. A RED # assertion is evidence only after the baseline passes. case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root|comment_id|flow_root) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -392,4 +392,31 @@ for sequence_item in '- item' ' - item' '-'; do done fi + + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == comment_id ]]; then +cat > "$work/comment-id.md" <<'DOC' + + + +--- +id: # deliberately absent +type: Feature +--- +A preserved body paragraph. +DOC +cp "$work/comment-id.md" "$work/comment-id.original" +bash "$checker" --check "$work/comment-id.md" +bash "$checker" "$work/comment-id.md" +cmp "$work/comment-id.original" "$work/comment-id.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == flow_root ]]; then +for flow_root in '[one, two]' '{key: value}'; do + printf '%s\n' '---' "$flow_root" > "$work/flow-root.md" + cp "$work/flow-root.md" "$work/flow-root.original" + if bash "$checker" --check "$work/flow-root.md"; then exit 1; fi + if bash "$checker" "$work/flow-root.md"; then exit 1; else test "$?" = 1; fi + cmp "$work/flow-root.original" "$work/flow-root.md" +done +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' From ca0e02df3ec4e42879fb1c52bb866708b4e64caf Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 13:57:22 -0700 Subject: [PATCH 24/27] docs(planning): record completed card evidence and execution boundaries --- tasks/K01.md | 5 ++++- tasks/K02.md | 2 +- tasks/K03.md | 5 ++++- tasks/S01.md | 19 ++++++++++++++----- tasks/S02.md | 19 ++++++++++++++----- tasks/S03.md | 19 ++++++++++++++----- tasks/S04.md | 16 +++++++++++----- 7 files changed, 62 insertions(+), 23 deletions(-) diff --git a/tasks/K01.md b/tasks/K01.md index 6b98b7a1..df24a622 100644 --- a/tasks/K01.md +++ b/tasks/K01.md @@ -7,6 +7,9 @@ tracking_issue: https://github.com/flyingrobots/echo/issues/722 echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 depends_on: [] +status: complete +pr: https://github.com/flyingrobots/echo/pull/768 +integration_commit: bb20c57345374f476fa938789294a0890341eadd --- @@ -14,7 +17,7 @@ depends_on: [] # Feature ```text -Execute task K01 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +Task K01 completed in PR #768 at bb20c57345374f476fa938789294a0890341eadd. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. ``` ## 1. Background Context diff --git a/tasks/K02.md b/tasks/K02.md index 1cb4331d..59d72066 100644 --- a/tasks/K02.md +++ b/tasks/K02.md @@ -17,7 +17,7 @@ depends_on: [] # Feature ```text -Execute task K02 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +Task K02 completed in PR #769 at 2056c95fb891125cbfcc8405e438542e020b7f4c. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. ``` ## 1. Background Context diff --git a/tasks/K03.md b/tasks/K03.md index db5df376..4c9f9fa5 100644 --- a/tasks/K03.md +++ b/tasks/K03.md @@ -7,6 +7,9 @@ tracking_issue: https://github.com/flyingrobots/echo/issues/722 echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 depends_on: [K01, K02] +status: complete +pr: https://github.com/flyingrobots/echo/pull/770 +integration_commit: fe8789263a26fbcb7c7554c2b48f9c33b812c7eb --- @@ -14,7 +17,7 @@ depends_on: [K01, K02] # Feature ```text -Execute task K03 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +Task K03 completed in PR #770 at fe8789263a26fbcb7c7554c2b48f9c33b812c7eb. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. ``` ## 1. Background Context diff --git a/tasks/S01.md b/tasks/S01.md index 9e614c68..2cb946ca 100644 --- a/tasks/S01.md +++ b/tasks/S01.md @@ -5,6 +5,9 @@ title: Describe the reachable-state boundary of WorldlineState::state_root issue: https://github.com/flyingrobots/echo/issues/754 evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 depends_on: [] +status: complete +pr: https://github.com/flyingrobots/echo/pull/762 +integration_commit: da929ca6093977e909af20ef918e2431ad9b338c --- @@ -12,7 +15,7 @@ depends_on: [] # Feature ```text -Execute https://github.com/flyingrobots/echo/issues/754. Verify the cited source. Run the named witness in the shared guarded Docker worker. Record RED evidence. Apply the smallest repair. Update current documents. Run relevant checks. Commit and push the repair. Open one PR. Obtain current-head Code Lawyer and agy approval. Merge only after all required gates pass. Preserve causal authority and unrelated work. Stop when the acceptance criteria pass. +Task S01 completed in PR #762 at da929ca6093977e909af20ef918e2431ad9b338c. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. ``` ## 1. Background Context @@ -25,16 +28,22 @@ The original experiment files are absent. The reported timings and counts are so ## 2. Problem Description +Historical defect at the audited revision: + The API calls state_root() a full-state hash. The implementation excludes nodes that the root cannot reach. Create-if-absent produces such nodes. +Current result: The API and owning documents state the reachable-state boundary. Detached writes remain bound by patch and commit identities; hash bytes are unchanged. + ## 3. Proposed Solution +The merged repair followed this original scope: + Correct the API contract and the operation documentation. Preserve the version-1 hash law. Explain that patch and commit identities retain detached writes. ## 4. Prerequisites -- [ ] Use the shared guarded Docker worker for executable checks. -- [ ] Record RED evidence before the code repair, or before/after evidence for a documentation repair. +- [x] Use the shared guarded Docker worker for executable checks. +- [x] Record RED evidence before the code repair, or before/after evidence for a documentation repair. No other PR must merge first. Execution order does not establish a dependency. @@ -50,7 +59,7 @@ The API states the reachable-state boundary. `docs/architecture/application-cont ## 7. Definition of Done -The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact PR head. The PR merges with issue linkage. +Completed in [PR #762](https://github.com/flyingrobots/echo/pull/762), integrated at `da929ca6093977e909af20ef918e2431ad9b338c`. The named witness, relevant checks and current-head Code Lawyer and agy review passed. GitHub owns any reopened or follow-on work. ## 8. Test Plan @@ -58,7 +67,7 @@ Golden: Inspect the old and new API text. Run a focused warp-core test that crea Edges: Reachable nodes, detached nodes, descended instances, and duplicate no-mutation evidence. -Known failure modes: The API calls state_root() a full-state hash. The implementation excludes nodes that the root cannot reach. Create-if-absent produces such nodes. +Known failure modes at the audited revision: The API calls state_root() a full-state hash. The implementation excludes nodes that the root cannot reach. Create-if-absent produces such nodes. Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. diff --git a/tasks/S02.md b/tasks/S02.md index 2501cd87..d6d6bed8 100644 --- a/tasks/S02.md +++ b/tasks/S02.md @@ -5,6 +5,9 @@ title: Run xtask run-edict-operation without a Git checkout or directory change issue: https://github.com/flyingrobots/echo/issues/755 evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 depends_on: [] +status: complete +pr: https://github.com/flyingrobots/echo/pull/765 +integration_commit: 6ef53c42db04ef16fae9e90e847203453a211af3 --- @@ -12,7 +15,7 @@ depends_on: [] # Feature ```text -Execute https://github.com/flyingrobots/echo/issues/755. Verify the cited source. Run the named witness in the shared guarded Docker worker. Record RED evidence. Apply the smallest repair. Update current documents. Run relevant checks. Commit and push the repair. Open one PR. Obtain current-head Code Lawyer and agy approval. Merge only after all required gates pass. Preserve causal authority and unrelated work. Stop when the acceptance criteria pass. +Task S02 completed in PR #765 at 6ef53c42db04ef16fae9e90e847203453a211af3. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. ``` ## 1. Background Context @@ -25,16 +28,22 @@ The original experiment files are absent. The reported timings and counts are so ## 2. Problem Description +Historical defect at the audited revision: + main() resolves a Git root and changes the current directory before it parses the command. The operation runner needs supplied artifact paths. It fails outside Git and resolves relative paths from a different directory inside Git. +Current result: The operation runner parses first and uses caller-relative paths without Git-root discovery or a directory change. Repository maintenance commands retain their root behavior. + ## 3. Proposed Solution +The merged repair followed this original scope: + Parse the command first. Exempt run-edict-operation from both Git-root discovery and the directory change. Keep both steps for repository commands. ## 4. Prerequisites -- [ ] Use the shared guarded Docker worker for executable checks. -- [ ] Record RED evidence before the code repair, or before/after evidence for a documentation repair. +- [x] Use the shared guarded Docker worker for executable checks. +- [x] Record RED evidence before the code repair, or before/after evidence for a documentation repair. No other PR must merge first. Execution order does not establish a dependency. @@ -50,7 +59,7 @@ The runner succeeds outside Git with absolute paths. It resolves relative artifa ## 7. Definition of Done -The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact PR head. The PR merges with issue linkage. +Completed in [PR #765](https://github.com/flyingrobots/echo/pull/765), integrated at `6ef53c42db04ef16fae9e90e847203453a211af3`. The named witness, relevant checks and current-head Code Lawyer and agy review passed. GitHub owns any reopened or follow-on work. ## 8. Test Plan @@ -58,7 +67,7 @@ Golden: `cargo test -p xtask --test run_edict_operation`. Add cases for a direct Edges: Absolute paths, relative paths, nested directories, malformed inputs, and repository commands. -Known failure modes: main() resolves a Git root and changes the current directory before it parses the command. The operation runner needs supplied artifact paths. It fails outside Git and resolves relative paths from a different directory inside Git. +Known failure modes at the audited revision: main() resolves a Git root and changes the current directory before it parses the command. The operation runner needs supplied artifact paths. It fails outside Git and resolves relative paths from a different directory inside Git. Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. diff --git a/tasks/S03.md b/tasks/S03.md index 5428096f..6927db6b 100644 --- a/tasks/S03.md +++ b/tasks/S03.md @@ -5,6 +5,9 @@ title: Report the typed Action obstruction when the operation runner cannot comm issue: https://github.com/flyingrobots/echo/issues/756 evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 depends_on: [] +status: complete +pr: https://github.com/flyingrobots/echo/pull/766 +integration_commit: 18b22e362e986f3e2509856433d040f33dc81bd2 --- @@ -12,7 +15,7 @@ depends_on: [] # Feature ```text -Execute https://github.com/flyingrobots/echo/issues/756. Verify the cited source. Run the named witness in the shared guarded Docker worker. Record RED evidence. Apply the smallest repair. Update current documents. Run relevant checks. Commit and push the repair. Open one PR. Obtain current-head Code Lawyer and agy approval. Merge only after all required gates pass. Preserve causal authority and unrelated work. Stop when the acceptance criteria pass. +Task S03 completed in PR #766 at 18b22e362e986f3e2509856433d040f33dc81bd2. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. ``` ## 1. Background Context @@ -25,16 +28,22 @@ The original experiment files are absent. The reported timings and counts are so ## 2. Problem Description +Historical defect at the audited revision: + The runner discards a non-committed Action outcome. It reports only that the scheduler did not publish a committed outcome. ResultProjectionInvalid therefore disappears from the caller error. +Current result: Both Action failure paths report bounded typed outcome categories. Raw records, invocation data and child backtraces are omitted; retained obstruction encoding is unchanged. + ## 3. Proposed Solution +The merged repair followed this original scope: + Match the first Action outcome and the unexpected duplicate outcome explicitly. Use the same bounded formatter for both paths. Report an obstruction through its fieldless kind. Report footprint conflict with a fixed category and a bounded count. Report an absent outcome separately. Do not print full Debug records or invocation data. Keep retained obstruction encoding unchanged. ## 4. Prerequisites -- [ ] Use the shared guarded Docker worker for executable checks. -- [ ] Record RED evidence before the code repair, or before/after evidence for a documentation repair. +- [x] Use the shared guarded Docker worker for executable checks. +- [x] Record RED evidence before the code repair, or before/after evidence for a documentation repair. No other PR must merge first. Execution order does not establish a dependency. @@ -50,7 +59,7 @@ An output-bound refusal names ResultProjectionInvalid. Each outcome summary stay ## 7. Definition of Done -The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact PR head. The PR merges with issue linkage. +Completed in [PR #766](https://github.com/flyingrobots/echo/pull/766), integrated at `18b22e362e986f3e2509856433d040f33dc81bd2`. The named witness, relevant checks and current-head Code Lawyer and agy review passed. GitHub owns any reopened or follow-on work. ## 8. Test Plan @@ -58,7 +67,7 @@ Golden: `cargo test -p xtask --test run_edict_operation`. Use a canonical input Edges: Missing outcome, projection refusal, footprint conflict, committed outcome, and duplicate obstruction. Check the message length and absence of raw record fields. -Known failure modes: The runner discards a non-committed Action outcome. It reports only that the scheduler did not publish a committed outcome. ResultProjectionInvalid therefore disappears from the caller error. +Known failure modes at the audited revision: The runner discards a non-committed Action outcome. It reports only that the scheduler did not publish a committed outcome. ResultProjectionInvalid therefore disappears from the caller error. Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. diff --git a/tasks/S04.md b/tasks/S04.md index 605f21d7..04181bdd 100644 --- a/tasks/S04.md +++ b/tasks/S04.md @@ -15,7 +15,7 @@ depends_on: [] # Feature ```text -Execute https://github.com/flyingrobots/echo/issues/757. Verify the cited source. Run the named witness in the shared guarded Docker worker. Record RED evidence. Apply the smallest repair. Update current documents. Run relevant checks. Commit and push the repair. Open one PR. Obtain current-head Code Lawyer and agy approval. Merge only after all required gates pass. Preserve causal authority and unrelated work. Stop when the acceptance criteria pass. +Task S04 completed in PR #767 at 7dde48b223ed105c13a1b70afb6a1a07d1dc308e. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. ``` ## 1. Background Context @@ -28,10 +28,16 @@ The original experiment files are absent. The reported timings and counts are so ## 2. Problem Description +Historical defect at the audited revision: + Recovery caches a full WorldlineState for every distinct basis tick. Each cache miss replays from the initial boundary when no checkpoint exists. Consecutive bases repeat all earlier patches and retain growing graph and tick-history copies. +Current result: Recovery verifies exact obligations through ordered replay cursors and two sweeps, preserving retained protocol order. It retains one cursor per needed worldline and one transient Tick state. Counters describe replay-patch work and logical retained data, with no linear elapsed-time or process-RSS claim. + ## 3. Proposed Solution +The merged repair followed this original scope: + Index verification obligations by exact coordinate without changing retained protocol order. Use bounded replay cursors and the existing replay validation primitives. Preserve all basis, patch, receipt, decision, conflict, and initial-boundary checks. 1. Validate record identities and order under the existing laws. Index receipt and Action basis obligations by worldline and tick. Index composite Ticks separately by their parent coordinate. Store references and compact preparation evidence, not full states. @@ -45,8 +51,8 @@ Bound replay-patch applications by two verified history sweeps per parent-state ## 4. Prerequisites -- [ ] Use the shared guarded Docker worker for executable checks. -- [ ] Record RED evidence before the code repair, or before/after evidence for a documentation repair. +- [x] Use the shared guarded Docker worker for executable checks. +- [x] Record RED evidence before the code repair, or before/after evidence for a documentation repair. No other PR must merge first. Execution order does not establish a dependency. @@ -62,7 +68,7 @@ A multi-tick recovery witness counts applied replay patches and retained replay ## 7. Definition of Done -The witness passes. Relevant checks pass. Code Lawyer and agy approve the exact PR head. The PR merges with issue linkage. +Completed in [PR #767](https://github.com/flyingrobots/echo/pull/767), integrated at `7dde48b223ed105c13a1b70afb6a1a07d1dc308e`. The named witness, relevant checks and current-head Code Lawyer and agy review passed. GitHub owns any reopened or follow-on work. ## 8. Test Plan @@ -70,7 +76,7 @@ Golden: `cargo test -p warp-core --features native_rule_bootstrap,trusted_runtim Edges: Shared bases, delayed stale bases, non-monotonic retained basis order, cross-worldline bases, multiple heads, checkpoints, batches, conflicts, and corrupted retained evidence. Compare accepted and refused postures with the old validator. -Known failure modes: Recovery caches a full WorldlineState for every distinct basis tick. Each cache miss replays from the initial boundary when no checkpoint exists. Consecutive bases repeat all earlier patches and retain growing graph and tick-history copies. +Known failure modes at the audited revision: Recovery caches a full WorldlineState for every distinct basis tick. Each cache miss replays from the initial boundary when no checkpoint exists. Consecutive bases repeat all earlier patches and retain growing graph and tick-history copies. Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. From f48b71237b6f693dbcd06aa22563e4710c4ae2cc Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 14:13:15 -0700 Subject: [PATCH 25/27] fix(docs): preserve fenced examples and replace reserved copyright attempts --- docs/DOCUMENTATION_STANDARDS.md | 2 +- scripts/ensure_spdx.sh | 8 ++++++-- scripts/tests/spdx_frontmatter_test.sh | 24 +++++++++++++++++++++++- tasks/K02.md | 4 ++-- 4 files changed, 32 insertions(+), 6 deletions(-) diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index b52b9f33..7a6b05b8 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,7 +163,7 @@ when one exists. ## Writing and structure -Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key, block-sequence or flow-container hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. +Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key, block-sequence or flow-container hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`, with no fence-starter lines in the candidate block. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index a0e52641..6102d0ba 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -109,7 +109,7 @@ get_header_content() { # Only complete HTML license/copyright comments belong to a Markdown header. MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*$' -MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*$' +MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*$' markdown_metadata_bounds() { awk -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' @@ -155,8 +155,12 @@ markdown_metadata_bounds() { finish = 0 has_id = 0 has_type = 0 + has_fenced_example = 0 for (i = start + 1; i <= NR; i++) { if (lines[i] == "---") { finish = i; break } + # A fenced example makes a legacy candidate ambiguous. Preserve it + # rather than interpreting example id/type lines as live task keys. + if (lines[i] ~ /^[[:space:]]*```/ || lines[i] ~ /^[[:space:]]*~~~/) has_fenced_example = 1 if (lines[i] ~ /^id[[:space:]]*:/) { id_value = lines[i] sub(/^id[[:space:]]*:[[:space:]]*/, "", id_value) @@ -183,7 +187,7 @@ markdown_metadata_bounds() { } # Only the explicit task-card schema authorizes automatic relocation. # Generic body prose after a licensed header is never moved on key: alone. - if (start > 1 && !(has_id && has_type)) { print 0, 0; exit } + if (start > 1 && !(has_id && has_type && !has_fenced_example)) { print 0, 0; exit } print start, finish } ' "$1" diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 27aaebe4..cdb688e7 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -9,7 +9,7 @@ checker=${1:-scripts/ensure_spdx.sh} # baseline. Selectors narrow the added regression, not this preflight. A RED # assertion is evidence only after the baseline passes. case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root|comment_id|flow_root) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root|comment_id|flow_root|fenced_task|copyright_attempt) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -419,4 +419,26 @@ for flow_root in '[one, two]' '{key: value}'; do cmp "$work/flow-root.original" "$work/flow-root.md" done fi + + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == fenced_task ]]; then +for fence in '```' '````' '~~~' '~~~~'; do + { + printf '%s\n' '' '' '' '---' 'Note: task card example.' + printf '%syaml\n' "$fence" + printf '%s\n' 'id: S01' 'type: Feature' "$fence" '---' 'Preserved body.' + } > "$work/fenced-task.md" + cp "$work/fenced-task.md" "$work/fenced-task.original" + bash "$checker" --check "$work/fenced-task.md" + bash "$checker" "$work/fenced-task.md" + cmp "$work/fenced-task.original" "$work/fenced-task.md" +done +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == copyright_attempt ]]; then +printf '%s\n' '' '' '# Preserved body' > "$work/copyright-attempt.md" +if bash "$checker" "$work/copyright-attempt.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/copyright-attempt.md" +test "$(grep -c '© James Ross' "$work/copyright-attempt.md")" = 1 +grep -qx '# Preserved body' "$work/copyright-attempt.md" +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' diff --git a/tasks/K02.md b/tasks/K02.md index 59d72066..9ed62067 100644 --- a/tasks/K02.md +++ b/tasks/K02.md @@ -38,7 +38,7 @@ Add an Echo-owned fallible complete-object port. Implement MemoryTier and DiskTi ## 4. Prerequisites -- [ ] Recheck the Echo physical-content boundary and existing CAS adapter contracts. +- [x] Recheck the Echo physical-content boundary and existing CAS adapter contracts. Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. @@ -54,7 +54,7 @@ Existing backends pass the same complete-object tests. No failed read exposes a ## 7. Definition of Done -The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. +Completed in [PR #769](https://github.com/flyingrobots/echo/pull/769), integrated at `2056c95fb891125cbfcc8405e438542e020b7f4c`. The shared complete-object suite, 36 existing CAS tests, relevant strict checks and current-head Code Lawyer and agy reviews passed. Existing consumers remain compatible; initial receipts establish no durability or authenticated absence. ## 8. Test Plan From 374ef342f62776fe18bbf9ed788fcfc2cf001e03 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 14:30:05 -0700 Subject: [PATCH 26/27] fix(docs): admit scalar task IDs and preserve delimiter whitespace --- docs/DOCUMENTATION_STANDARDS.md | 2 +- scripts/ensure_spdx.sh | 19 +++++++++--- scripts/tests/spdx_frontmatter_test.sh | 41 +++++++++++++++++++++++++- tasks/K02.md | 2 +- tasks/S02.md | 2 +- 5 files changed, 58 insertions(+), 8 deletions(-) diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index 7a6b05b8..f0fae01c 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,7 +163,7 @@ when one exists. ## Writing and structure -Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key, block-sequence or flow-container hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level nonempty `id` and recognized `type`, with no fence-starter lines in the candidate block. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. +Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block (optional trailing spaces or tabs are structural) as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key, block-sequence or flow-container hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level simple nonempty `id` scalar and recognized `type`, with no fence-starter lines in the candidate block. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index 6102d0ba..614f8967 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -126,7 +126,12 @@ markdown_metadata_bounds() { first_char = substr(line, 1, 1) return map_key(line) || line ~ /^\?[[:space:]]/ || line ~ /^-([[:space:]]|$)/ || first_char == "[" || first_char == "{" } - { parsed = $0; sub(/\r$/, "", parsed); lines[NR] = parsed } + { + parsed = $0 + sub(/\r$/, "", parsed) + if (parsed ~ /^---[[:blank:]]*$/) parsed = "---" + lines[NR] = parsed + } END { start = 0 if (lines[1] == "---") start = 1 @@ -168,10 +173,16 @@ markdown_metadata_bounds() { sub(/[[:space:]]+#.*$/, "", id_value) sub(/[[:space:]]*$/, "", id_value) id_quote = substr(id_value, 1, 1) - if ((id_quote == "\"" || id_quote == sprintf("%c", 39)) && substr(id_value, length(id_value), 1) == id_quote) { - id_value = substr(id_value, 2, length(id_value) - 2) + if (id_quote == "\"" || id_quote == sprintf("%c", 39)) { + if (length(id_value) >= 2 && substr(id_value, length(id_value), 1) == id_quote) { + id_value = substr(id_value, 2, length(id_value) - 2) + if (id_value != "") has_id = 1 + } + } else if (id_value != "" && tolower(id_value) != "null" && id_value != "~" && index("[]{}|>&*!", id_quote) == 0) { + # Only a simple nonempty scalar authorizes legacy relocation. + # Collections, block scalars, aliases and tags need manual placement. + has_id = 1 } - if (id_value != "" && id_value != "null" && id_value != "~") has_id = 1 } if (lines[i] ~ /^type[[:space:]]*:/) { value = lines[i] diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index cdb688e7..995f9445 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -9,7 +9,7 @@ checker=${1:-scripts/ensure_spdx.sh} # baseline. Selectors narrow the added regression, not this preflight. A RED # assertion is evidence only after the baseline passes. case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root|comment_id|flow_root|fenced_task|copyright_attempt) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root|comment_id|flow_root|fenced_task|copyright_attempt|empty_yaml_id|spaced_delimiters) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -441,4 +441,43 @@ bash "$checker" --check "$work/copyright-attempt.md" test "$(grep -c '© James Ross' "$work/copyright-attempt.md")" = 1 grep -qx '# Preserved body' "$work/copyright-attempt.md" fi + + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == empty_yaml_id ]]; then +for id_value in NULL Null '[]' '{}'; do + printf '%s\n' '' '' '' '---' "id: $id_value" 'type: Feature' '---' 'Preserved body.' > "$work/empty-yaml-id.md" + cp "$work/empty-yaml-id.md" "$work/empty-yaml-id.original" + bash "$checker" --check "$work/empty-yaml-id.md" + bash "$checker" "$work/empty-yaml-id.md" + cmp "$work/empty-yaml-id.original" "$work/empty-yaml-id.md" +done +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == spaced_delimiters ]]; then +for shape in opening closing both; do + opening='---';closing='---' + if [[ "$shape" != closing ]]; then opening='--- '; fi + if [[ "$shape" != opening ]]; then closing=$'---\t'; fi + printf '%s\n' "$opening" 'id: S01' "$closing" '# Body' > "$work/spaced-delimiters.md" + head -n 3 "$work/spaced-delimiters.md" > "$work/spaced-delimiters.original" + if bash "$checker" "$work/spaced-delimiters.md"; then exit 1; else test "$?" = 1; fi + bash "$checker" --check "$work/spaced-delimiters.md" + head -n 3 "$work/spaced-delimiters.md" > "$work/spaced-delimiters.actual" + cmp "$work/spaced-delimiters.original" "$work/spaced-delimiters.actual" +done +fi + + +# Closed companion controls preserve the deliberate unclosed refusal witnesses. +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == sequence_root || "${SPDX_CASE:-all}" == flow_root ]]; then +for root_value in '- item' ' - item' '-' '[one, two]' '{key: value}'; do + printf '%s\n' '---' "$root_value" '---' > "$work/closed-root.md" + cp "$work/closed-root.md" "$work/closed-root.original" + if bash "$checker" "$work/closed-root.md"; then exit 1; else test "$?" = 1; fi + bash "$checker" --check "$work/closed-root.md" + head -n 3 "$work/closed-root.md" > "$work/closed-root.actual" + cmp "$work/closed-root.original" "$work/closed-root.actual" + test "$(sed -n '4p' "$work/closed-root.md")" = '' + test "$(sed -n '5p' "$work/closed-root.md")" = '' +done +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' diff --git a/tasks/K02.md b/tasks/K02.md index 9ed62067..88b70ca1 100644 --- a/tasks/K02.md +++ b/tasks/K02.md @@ -54,7 +54,7 @@ Existing backends pass the same complete-object tests. No failed read exposes a ## 7. Definition of Done -Completed in [PR #769](https://github.com/flyingrobots/echo/pull/769), integrated at `2056c95fb891125cbfcc8405e438542e020b7f4c`. The shared complete-object suite, 36 existing CAS tests, relevant strict checks and current-head Code Lawyer and agy reviews passed. Existing consumers remain compatible; initial receipts establish no durability or authenticated absence. +Completed in [PR #769](https://github.com/flyingrobots/echo/pull/769), integrated at `2056c95fb891125cbfcc8405e438542e020b7f4c`. The final `k02-fresh-fixture-gate` passed 36 CAS tests (17 unit, four disk, five physical-content and ten retention), with CAS source hashes matching the merged candidate. The initial gate had 34 before later regression tests were added. Relevant strict checks and current-head Code Lawyer and agy reviews passed. Existing consumers remain compatible; initial receipts establish no durability or authenticated absence. ## 8. Test Plan diff --git a/tasks/S02.md b/tasks/S02.md index d6d6bed8..a192c2aa 100644 --- a/tasks/S02.md +++ b/tasks/S02.md @@ -63,7 +63,7 @@ Completed in [PR #765](https://github.com/flyingrobots/echo/pull/765), integrate ## 8. Test Plan -Golden: `cargo test -p xtask --test run_edict_operation`. Add cases for a directory outside Git and a nested directory inside an unrelated repository. +Golden: `cargo test -p xtask --test run_edict_operation`. Covered by `runner_accepts_absolute_artifacts_outside_git` and `runner_preserves_relative_paths_in_an_unrelated_nested_repository`. Edges: Absolute paths, relative paths, nested directories, malformed inputs, and repository commands. From 826172cee2fa8273e9e4ae9eb8021d318a61a3a0 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 7 Oct 2026 14:38:59 -0700 Subject: [PATCH 27/27] fix(docs): require explicit line-one metadata and preserve legacy bodies --- docs/DOCUMENTATION_STANDARDS.md | 2 +- scripts/ensure_spdx.sh | 75 ++++---------------------- scripts/tests/spdx_frontmatter_test.sh | 49 ++++++++++------- 3 files changed, 40 insertions(+), 86 deletions(-) diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index f0fae01c..c12b390e 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -163,7 +163,7 @@ when one exists. ## Writing and structure -Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block (optional trailing spaces or tabs are structural) as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key, block-sequence or flow-container hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. For a legacy header-first file, automatic relocation is limited to task metadata with an unindented top-level simple nonempty `id` scalar and recognized `type`, with no fence-starter lines in the candidate block. Place other displaced frontmatter manually: generic map-like text after a licensed header is ambiguous with body prose, so the checker preserves it. +Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block (optional trailing spaces or tabs are structural) as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key, block-sequence or flow-container hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. The checker does not infer metadata from licensed body sections or relocate them. Place intended legacy frontmatter manually at line one. Task-like fields, quoted values and fenced examples can also be body prose; their presence cannot establish author intent. A successful license check certifies header placement relative to recognized line-one framing, not all metadata or body semantics. - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index 614f8967..4b76cae6 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -112,7 +112,7 @@ MD_LICENSE_COMMENT_PATTERN='^[[: MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*$' markdown_metadata_bounds() { - awk -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + awk ' function map_key(line, quote) { quote = sprintf("%c", 39) trimmed = line @@ -133,73 +133,20 @@ markdown_metadata_bounds() { lines[NR] = parsed } END { - start = 0 - if (lines[1] == "---") start = 1 - else { - saw_header = 0 - for (i = 1; i <= NR; i++) { - if (lines[i] ~ license_re || lines[i] ~ copyright_re) { saw_header = 1; continue } - if (lines[i] ~ /^[[:space:]]*$/) continue - if (saw_header && lines[i] == "---") start = i - break - } - } - if (!start) { print 0, 0; exit } + # Delimiters at line one declare the author-selected metadata aperture. + # Licensed body sections are ambiguous and are never relocated. + if (lines[1] != "---") { print 0, 0; exit } # A complete line-one delimiter block is metadata; YAML grammar belongs - # to its consumer. Legacy header-first relocation still needs task keys. - if (start == 1) { - for (i = 2; i <= NR; i++) { - if (lines[i] == "---") { print 1, i; exit } - } + # to its consumer. Original delimiter and payload bytes are preserved. + for (i = 2; i <= NR; i++) { + if (lines[i] == "---") { print 1, i; exit } } - first = start + 1 + first = 2 while (first <= NR && (lines[first] ~ /^[[:space:]]*$/ || lines[first] ~ /^[[:space:]]*#/)) first++ if (first > NR || !metadata_hint(lines[first])) { print 0, 0; exit } - finish = 0 - has_id = 0 - has_type = 0 - has_fenced_example = 0 - for (i = start + 1; i <= NR; i++) { - if (lines[i] == "---") { finish = i; break } - # A fenced example makes a legacy candidate ambiguous. Preserve it - # rather than interpreting example id/type lines as live task keys. - if (lines[i] ~ /^[[:space:]]*```/ || lines[i] ~ /^[[:space:]]*~~~/) has_fenced_example = 1 - if (lines[i] ~ /^id[[:space:]]*:/) { - id_value = lines[i] - sub(/^id[[:space:]]*:[[:space:]]*/, "", id_value) - if (id_value ~ /^#/) id_value = "" - sub(/[[:space:]]+#.*$/, "", id_value) - sub(/[[:space:]]*$/, "", id_value) - id_quote = substr(id_value, 1, 1) - if (id_quote == "\"" || id_quote == sprintf("%c", 39)) { - if (length(id_value) >= 2 && substr(id_value, length(id_value), 1) == id_quote) { - id_value = substr(id_value, 2, length(id_value) - 2) - if (id_value != "") has_id = 1 - } - } else if (id_value != "" && tolower(id_value) != "null" && id_value != "~" && index("[]{}|>&*!", id_quote) == 0) { - # Only a simple nonempty scalar authorizes legacy relocation. - # Collections, block scalars, aliases and tags need manual placement. - has_id = 1 - } - } - if (lines[i] ~ /^type[[:space:]]*:/) { - value = lines[i] - sub(/^type[[:space:]]*:[[:space:]]*/, "", value) - sub(/[[:space:]]+#.*$/, "", value) - sub(/[[:space:]]*$/, "", value) - quote = substr(value, 1, 1) - if ((quote == "\"" || quote == sprintf("%c", 39)) && substr(value, length(value), 1) == quote) { - value = substr(value, 2, length(value) - 2) - } - if (value ~ /^(Feature|Bug|Decision|Research|Investigation|Spike)$/) has_type = 1 - } - } - # Only the explicit task-card schema authorizes automatic relocation. - # Generic body prose after a licensed header is never moved on key: alone. - if (start > 1 && !(has_id && has_type && !has_fenced_example)) { print 0, 0; exit } - print start, finish + print 1, 0 } ' "$1" } @@ -227,8 +174,7 @@ check_valid_header() { local metadata_start metadata_end read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" if [[ "$metadata_start" -gt 0 ]]; then - # A header-first metadata block must move to the start in repair mode. - if [[ "$metadata_start" -ne 1 || "$metadata_end" -eq 0 ]]; then return 1; fi + if [[ "$metadata_end" -eq 0 ]]; then return 1; fi i=$metadata_end fi fi @@ -295,7 +241,6 @@ strip_existing_headers() { BEGIN { header_active = 1 } { # The bounds helper admits only header comments and whitespace here. - if (metadata_start > 1 && NR < metadata_start) next if (metadata_start > 0 && NR <= metadata_end) { print; next } if (header_active) { if (NR > metadata_end + 15) header_active = 0 diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh index 995f9445..89c7e967 100755 --- a/scripts/tests/spdx_frontmatter_test.sh +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -9,7 +9,7 @@ checker=${1:-scripts/ensure_spdx.sh} # baseline. Selectors narrow the added regression, not this preflight. A RED # assertion is evidence only after the baseline passes. case "${SPDX_CASE:-all}" in - all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root|comment_id|flow_root|fenced_task|copyright_attempt|empty_yaml_id|spaced_delimiters) ;; + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root|comment_id|flow_root|fenced_task|copyright_attempt|empty_yaml_id|spaced_delimiters|legacy_body) ;; *) echo 'unknown SPDX regression case' >&2; exit 2 ;; esac work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") @@ -82,20 +82,10 @@ type: Feature --- # Task DOC -if bash "$checker" --check "$work/header-first.md"; then - echo 'displaced metadata passed' >&2 - exit 1 -fi -if bash "$checker" "$work/header-first.md"; then - echo 'metadata relocation must report a change' >&2 - exit 1 -else - test "$?" = 1 -fi +cp "$work/header-first.md" "$work/header-first.original" bash "$checker" --check "$work/header-first.md" -head -n 4 "$work/valid.md" > "$work/metadata.expected" -head -n 4 "$work/header-first.md" > "$work/metadata.actual" -cmp "$work/metadata.expected" "$work/metadata.actual" +bash "$checker" "$work/header-first.md" +cmp "$work/header-first.original" "$work/header-first.md" fi if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == displaced ]]; then cat > "$work/displaced.md" <<'DOC' @@ -313,10 +303,10 @@ type: "Feature" --- # Body DOC -if bash "$checker" --check "$work/quoted-type.md"; then exit 1; else test "$?" = 1; fi -if bash "$checker" "$work/quoted-type.md"; then exit 1; else test "$?" = 1; fi +cp "$work/quoted-type.md" "$work/quoted-type.original" bash "$checker" --check "$work/quoted-type.md" -test "$(head -n 1 "$work/quoted-type.md")" = '---' +bash "$checker" "$work/quoted-type.md" +cmp "$work/quoted-type.original" "$work/quoted-type.md" fi if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == crlf ]]; then printf -- '---\r\nid: S01\r\n---\r\n# Body\r\n' > "$work/crlf.md" @@ -376,10 +366,10 @@ type: Feature # task category --- # Body DOC -if bash "$checker" --check "$work/type-comment.md"; then exit 1; else test "$?" = 1; fi -if bash "$checker" "$work/type-comment.md"; then exit 1; else test "$?" = 1; fi +cp "$work/type-comment.md" "$work/type-comment.original" bash "$checker" --check "$work/type-comment.md" -test "$(head -n 1 "$work/type-comment.md")" = '---' +bash "$checker" "$work/type-comment.md" +cmp "$work/type-comment.original" "$work/type-comment.md" fi if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == sequence_root ]]; then @@ -480,4 +470,23 @@ for root_value in '- item' ' - item' '-' '[one, two]' '{key: value}'; do test "$(sed -n '5p' "$work/closed-root.md")" = '' done fi + + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == legacy_body ]]; then +cat > "$work/legacy-body.md" <<'DOC' + + + +--- +Note: a body section can contain task-like fields. +id: S01 +type: Feature +--- +Preserved body. +DOC +cp "$work/legacy-body.md" "$work/legacy-body.original" +bash "$checker" --check "$work/legacy-body.md" +bash "$checker" "$work/legacy-body.md" +cmp "$work/legacy-body.original" "$work/legacy-body.md" +fi printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation'