diff --git a/.github/workflows/spdx-header-check.yml b/.github/workflows/spdx-header-check.yml index 22c01833..eadaf3e5 100644 --- a/.github/workflows/spdx-header-check.yml +++ b/.github/workflows/spdx-header-check.yml @@ -26,6 +26,9 @@ jobs: chmod +x check_spdx.sh fi + - name: Verify Markdown license preservation + run: bash scripts/tests/spdx_frontmatter_test.sh + - name: Run SPDX check run: | if [ -f "scripts/check_spdx.sh" ]; then diff --git a/AGENTS.md b/AGENTS.md index f47daf32..7ba7fe2f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,6 +83,11 @@ issue or pull request. Record a durable decision in the named current document that owns its concept, and state supersession, refinement, dependency, and related-decision edges explicitly. Do not allocate a new numbered ADR. +The user-requested Echo study-feedback `ROADMAP.md` and linked `tasks/` cards +are a scoped exception to the checked-in-plan restriction. They project the +requested work; GitHub still owns issue status and accepted dependencies. +This exception does not authorize production Keep adoption or other backlogs. + When recovering context, read the relevant canonical topic/spec/invariant and architecture document, follow any explicit links into the historical ADR archive, then inspect the current GitHub issue or pull request, `git log -n 5`, diff --git a/CHANGELOG.md b/CHANGELOG.md index 46b204e3..30bfbeb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,8 @@ ### Fixed +- The SPDX checker preserves Markdown frontmatter when it checks or repairs license headers. + - Action WAL recovery reuses ordered verified replay cursors instead of retaining a full state for each basis tick. It preserves basis, Tick, result, obstruction and conflict checks, including delayed stale bases. - The generic operation runner names typed obstruction kinds, footprint conflicts, and missing outcomes in bounded summaries. It omits raw outcome records and invocation data on both Action error paths. diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 00000000..d6477af1 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,110 @@ + + + +# Echo Study Feedback Roadmap + +## Executive summary + +This plan addresses four defects confirmed in Echo source. The audit uses commit `a93e9d82e89455ed1fa0b63447c88de544b9da26`. + +The state-root API overstates its hash boundary. The operation runner requires an unrelated Git checkout and changes its input directory. Its error hides typed Action outcomes. Recovery repeats replay prefixes and retains each basis state. + +These repairs make evidence boundaries clear, make the supplied runner usable, and remove repeated recovery work. They do not complete every proposed Echo feature. The separate Keep sequence below supplies the requested CAS integration plan. + +The user requested this checked-in plan. That request overrides the normal policy that live plans exist only in GitHub. GitHub issues and PRs remain the status authority. + +The S01 repair landed in [PR #762](https://github.com/flyingrobots/echo/pull/762) at main commit `da929ca6093977e909af20ef918e2431ad9b338c`. GitHub records its final status. + +## Source and verification boundary + +Source: `FEEDBACK-echo.md`, SHA-256 `a831edf49300065e982f166dae8fd2a801f533c0e88fd8afb3e52108725ccdfa`. + +Reader source: `library/causal-computing/2026-10-07-synapse-backend-experiments/originals/FEEDBACK-echo.md`. + +The auditor read all 13 feedback sections and the affected Echo code. The original benchmark sources, WAL stores, and raw results are absent. Reported measurements remain unverified. Source inspection confirms mechanisms; it does not reproduce elapsed times. + +## Claim decisions + +| Item | Decision | Evidence and reason | +| --- | --- | --- | +| 1 | Confirmed documentation defect; S01 | `snapshot.rs:81-158` hashes reachable nodes. `echo_operation.rs:4651-4683` creates a node and attachment without an edge. `worldline_state.rs:249` incorrectly says full-state. The Merkle specification deliberately defines reachable state. Preserve that law. The patch digest binds detached writes. | +| 2 | Confirmed repeated work and retention; S04 | `trusted_runtime_host.rs:4196-4217` caches each basis state. `provenance_store.rs:1053-1077` restores a base and replays its prefix. Validation requests each distinct basis. However, `tick_history` contains snapshots, receipts, and patches, not full graph stores. The proposed snapshot explanation is incorrect. | +| 3 | Confirmed durability mechanism; no demonstrated latency defect | `submit_intent_with_runtime_wal_ack_inner` commits acceptance before return. `causal_wal.rs:5895,7860` syncs the commit file. Tick batching does not batch submission commits. The API already defines this durable ACK contract. The 20 ms figure is unverified. A batch API requires a separate accepted contract. | +| 4 | Unverified measurement; no stated size limit | WAL records retain submission, package, receipt, patch, and decision evidence. The report gives no permitted byte limit. Do not weaken retained evidence to meet an invented ratio. | +| 5 | Confirmed runner defect; S02 | `xtask/src/main.rs:454-461` discovers and enters a Git root before command parsing. The runner consumes supplied artifact paths. | +| 6 | Confirmed runner diagnostic defect; S03 | `run_edict_operation.rs:379-385` discards the outcome. `echo_operation.rs:4760-4765` retains a deterministic ResultProjectionInvalid kind. Expose that kind. New retained reason codes require a separate schema contract. | +| 7 | Confirmed declared codec boundary | `parse_input` checks shape and replacement bytes. README states that generic ingress does not validate codec-owned input schemas. The compiler cannot prove arbitrary caller JSON. Type enforcement belongs in the generated or authored adapter. No generic-core defect is established. | +| 8 | Confirmed bootstrap requirement | `build_host` reconstructs the initial lane. Replay validates the registered initial boundary. WAL docs describe bootstrap sources. A standalone open API is a feature proposal. Do not infer a broken recovery promise. | +| 9 | Broad claim rejected; narrower limitation confirmed | `TrustedRuntimeApp::observe` calls ObservationService. `ObservationAt::Tick` selects past coordinates and returns a reading envelope. `QueryBytes` optics remain unsupported. Bounded Edict setup hashes the frontier, as its current contract states. | +| 10 | Confirmed declared feature gap, already tracked | `WorldlineRuntime::fork_strand` exists. The trusted host has no durable fork crossing. WAL docs declare this limit. Existing issue #605 owns WAL-backed fork/drop work; do not create a duplicate or silently adopt that broader feature. | +| 11 | Confirmed host authority split | Cargo features distinguish internal rule authoring and trusted runtime ownership. Application handles cannot own scheduler authority. The suggested convenience crate is a product proposal. Its unavailable harness does not establish the claimed import count. | +| 12 | Pins confirmed; defect rejected | `hello-echo/producers.lock.json` pins exact producers. Its README requires those revisions and explains how pins advance. A refusal on another main revision enforces reproducibility. No automatic latest-main compatibility promise exists. | +| 13 | Source-reported praise | Existing runner tests cover these mechanisms. The reported 22,100 Actions and timings remain unverified without raw results. | + +Code paths are relative to this repository, except the explicitly named hello-echo consumer. All conclusions describe the audited revision. + +## Execution sequence + +- [x] [S01: Describe the reachable-state boundary of WorldlineState::state_root](tasks/S01.md) — [issue #754](https://github.com/flyingrobots/echo/issues/754) +- [x] [S02: Run xtask run-edict-operation without a Git checkout or directory change](tasks/S02.md) — [issue #755](https://github.com/flyingrobots/echo/issues/755) +- [x] [S03: Report the typed Action obstruction when the operation runner cannot commit](tasks/S03.md) — [issue #756](https://github.com/flyingrobots/echo/issues/756) +- [x] [S04: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery](tasks/S04.md) — [issue #757](https://github.com/flyingrobots/echo/issues/757) + +The original order was S01, S02, S03, then S04. Each repair landed in one independently mergeable PR. This order put smaller contract and runner repairs before recovery work. + +The feedback-repair graph has four vertices and zero required edges. All four tasks form one antichain at the audited revision. The chosen sequence serializes the shared worker; it does not imply a code prerequisite. + +Each task owns its stated defect. S01 owns hash-boundary text. S02 owns command directory behavior. S03 owns runner outcome errors. S04 owns recovery work and retention. No task owns a second task's requirements. + +## Keep CAS integration sequence + +Use `~/git/keep` as the source project. The local Keep checkout is `001ae2a`; its refreshed `origin/main` is `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. Inspect and pin the actual integration revision. Preserve the existing paused checkout. + +[Issue #722](https://github.com/flyingrobots/echo/issues/722) is the integration container. Its six milestones describe the same work as the six task documents below. Do not count the container as another executable PR. + +The [physical-content boundary](docs/architecture/echo-keep-physical-content-boundary.md) owns the accepted architecture. Keep supplies physical bytes and receipts. Echo retains content identity, WSC identity, causal history, semantic meaning, and authority. Keep receipts cannot replace Echo observations. + +Echo selects Rust 1.96.0 for its toolchain and warp-core. The workspace default and echo-cas still declare Rust 1.90.0. Keep requires Rust 1.96. The older Rust 1.90 statement in #722 is stale. A matching version number does not prove dependency, platform, or durability compatibility. + +Keep main exports `DurableStore` and fenced snapshots. Its crate documentation still states that production durable ingestion, garbage collection, and compaction are unimplemented. Thus durable reads and durable publication have different readiness. + +- [x] [K01: Prove the Echo and Keep content identity bridge](tasks/K01.md) — [issue #759](https://github.com/flyingrobots/echo/issues/759) +- [x] [K02: Add the Echo physical-content port and existing CAS adapters](tasks/K02.md) — [issue #760](https://github.com/flyingrobots/echo/issues/760) +- [x] [K03: Add an experimental Keep ReferenceStore adapter](tasks/K03.md) — [issue #761](https://github.com/flyingrobots/echo/issues/761) +- [ ] [K04: Add a pinned-generation durable Keep read adapter](tasks/K04.md) +- [ ] [K05: Prove durable Echo and Keep publication reconciliation](tasks/K05.md) +- [ ] [K06: Prepare the migration and production adoption decision](tasks/K06.md) + +K01 and K02 landed as independent slices, followed by K03. The experimental adapter passed its conformance and review gates in [PR #770](https://github.com/flyingrobots/echo/pull/770), integrated at `fe8789263a26fbcb7c7554c2b48f9c33b812c7eb`. This completes the authorized implementation scope. + +K04–K06 are conditional follow-on work. K04 requires guarded storage that actually passes Keep's Linux ext4 admission. K05 is blocked on external Keep durable ingestion, operation lookup, and non-expiring retention anchors. K06 also requires crash, migration, rollback, and adoption evidence. Completing K03 does not unblock these requirements. + +Proposed edges: K01 → K03; K02 → K03; K03 → K04; K04 → K05; K05 → K06. K01 supplies the identity law. K02 supplies the port. K03 supplies backend conformance. K04 supplies pinned-view receipt validation. K05 supplies durable reconciliation. + +No feedback repair requires Keep integration. No Keep integration task requires a feedback repair merely because both touch storage. The shared Docker worker requires serial execution, not a graph edge. + +The initial proposed graph has ten task vertices and five internal edges. S01, S02, S03, S04, K01, K02, and K03 are complete. The remaining conditional graph has three task vertices and two internal edges; its first structural layer is `{K04}`. No conditional task is ready for this run: external gates and the separate adoption decision remain unmet. The initial graph layers are `{S01,S02,S03,S04,K01,K02}`, `{K03}`, `{K04}`, `{K05}`, `{K06}`. External prerequisite nodes are `keep_durable_ingestion`, `durable_operation_lookup`, `non_expiring_retention_anchor`, and `admitted_guarded_storage`. Each has an unresolved edge into K05. Admitted guarded storage also gates K04. These nodes are capability requirements, not invented tracker issues. The ten-task, five-edge count covers internal task edges only. GitHub records #759 and #760 as blockers of #761 under container #722. Future K04–K06 edges remain proposals until their executable issues exist. Reconcile these tracker edges before execution. + +The user authorized K01–K03 for this run: implement the experimental adapter first. K04–K06 remain conditional follow-on work. The initial target is experimental integration. Production adoption requires a separately reviewable accepted decision after the evidence gates pass. Keep must not become the default through an incidental dependency change. + +## Solution constraints from independent critique + +S03 reports bounded outcome categories. It distinguishes a missing outcome from an obstruction and avoids raw record dumps. + +S04 indexes exact obligations before replay. It sorts verification references within each worldline and keeps retained protocol order authoritative. Its two-sweep plan supports stale and cross-worldline bases. Replay counters bound patch applications; they do not prove linear elapsed time. + +K01 verifies two distinct identity laws on the same source and reconstructed bytes. Equal source bytes do not make Echo and Keep digest values equal. K02 preserves existing CAS defaults and package compatibility. K03 keeps backend error causes private as well as binding fields: public formatting, downcast and source chains must not expose Keep coordinates. A late independent finding demonstrated this leak; the corrected adapter passed its regression and a fresh review. + +The independent critique confirms the S01 contract repair and experimental K01–K03 boundary. Its Reader source is receipt `7e05db3c-49d0-45df-b81b-2f1a6a25a6f7`. Reconciliation details remain on PR #758; this plan contains the resulting requirements. + +## Verification and merge rules + +Use the existing `echo-read-runtime` worker and stable `/lease-target` cache. Use `/Users/Shared/git-locks/workstation.git` with the concrete worker key `host/docker/echo-read-runtime/`. The monitor also inspects `echo-provider-builder`; reserve its key during validation. + +Keep the 20 GiB build, 4 GiB data, and 128 MiB log limits. Keep the 50 GiB host and VM free-space floors. Use copied source, four CPUs, 6 GiB RAM, bounded logs, process deadlines, and the fail-closed guard. Do not use host test fallbacks. + +Record before/after evidence for S01. Record executable RED/GREEN for behavior repairs. Use deterministic replay-work counters for S04. Do not claim a timing result from source inspection. + +Each PR must pass its relevant tests, linters, hosted required checks, and current-head Code Lawyer and agy reviews. Preserve issue, PR, and merge-commit linkage. Do not amend, rebase, or force-push. The user has authorized ordinary pushes and merges. + +STE prose follows the installed ASD-STE100 guide. Full dictionary conformance is unverified. diff --git a/docs/DOCUMENTATION_STANDARDS.md b/docs/DOCUMENTATION_STANDARDS.md index b0ed01c2..c12b390e 100644 --- a/docs/DOCUMENTATION_STANDARDS.md +++ b/docs/DOCUMENTATION_STANDARDS.md @@ -126,7 +126,7 @@ accepted contracts. GitHub owns change-local plans and status. Git history owns the exact old text. Do not check in backlogs, cycle packets, retrospectives, review transcripts, -status ledgers, or roadmap checklists. A short checked-in redirect may remain +status ledgers, or roadmap checklists. The user-requested Echo study-feedback ROADMAP and linked task cards are a scoped planning exception. GitHub remains their status and accepted-dependency authority. A short checked-in redirect may remain when an old stable path must route readers to its current owner. Historical reasoning must not masquerade as current behavior. Mark retained @@ -163,6 +163,8 @@ when one exists. ## Writing and structure +Markdown frontmatter remains first. The checker treats a complete line-one `---` delimiter block (optional trailing spaces or tabs are structural) as metadata and preserves its bytes; it does not validate YAML grammar. Unclosed line-one metadata with a mapping, explicit-key, block-sequence or flow-container hint, and unclosed reserved license-header attempts, refuse repair without mutation. Other ambiguous starter text is treated as body prose; this recognizer does not certify arbitrary YAML syntax. Place the SPDX and copyright comments immediately after its closing delimiter. The license checker validates this position when metadata starts at line one. The checker does not infer metadata from licensed body sections or relocate them. Place intended legacy frontmatter manually at line one. Task-like fields, quoted values and fenced examples can also be body prose; their presence cannot establish author intent. A successful license check certifies header placement relative to recognized line-one framing, not all metadata or body semantics. + - Lead with the result, decision, warning, or essential condition. - Prefer exact Echo terms and define unfamiliar ones at first use. - Use active voice when it clarifies ownership. diff --git a/scripts/ensure_spdx.sh b/scripts/ensure_spdx.sh index f3353e2c..4b76cae6 100755 --- a/scripts/ensure_spdx.sh +++ b/scripts/ensure_spdx.sh @@ -107,6 +107,50 @@ get_header_content() { esac } +# Only complete HTML license/copyright comments belong to a Markdown header. +MD_LICENSE_COMMENT_PATTERN='^[[:space:]]*$' +MD_COPYRIGHT_COMMENT_PATTERN='^[[:space:]]*$' + +markdown_metadata_bounds() { + awk ' + function map_key(line, quote) { + quote = sprintf("%c", 39) + trimmed = line + sub(/^[[:space:]]*/, "", trimmed) + if (substr(trimmed, 1, 1) == "\"") return trimmed ~ /^".*"[[:space:]]*:([[:space:]]|$)/ + if (substr(trimmed, 1, 1) == quote) return trimmed ~ ("^" quote ".*" quote "[[:space:]]*:([[:space:]]|$)") + return trimmed ~ /.+:([[:space:]]|$)/ + } + function metadata_hint(line, first_char) { + sub(/^[[:space:]]*/, "", line) + first_char = substr(line, 1, 1) + return map_key(line) || line ~ /^\?[[:space:]]/ || line ~ /^-([[:space:]]|$)/ || first_char == "[" || first_char == "{" + } + { + parsed = $0 + sub(/\r$/, "", parsed) + if (parsed ~ /^---[[:blank:]]*$/) parsed = "---" + lines[NR] = parsed + } + END { + # Delimiters at line one declare the author-selected metadata aperture. + # Licensed body sections are ambiguous and are never relocated. + if (lines[1] != "---") { print 0, 0; exit } + # A complete line-one delimiter block is metadata; YAML grammar belongs + # to its consumer. Original delimiter and payload bytes are preserved. + for (i = 2; i <= NR; i++) { + if (lines[i] == "---") { print 1, i; exit } + } + first = 2 + while (first <= NR && (lines[first] ~ /^[[:space:]]*$/ || lines[first] ~ /^[[:space:]]*#/)) first++ + if (first > NR || !metadata_hint(lines[first])) { + print 0, 0; exit + } + print 1, 0 + } + ' "$1" +} + check_valid_header() { local f="$1" local expected_header_block="$2" # This is a multi-line string like "# SPDX...\n# ©..." @@ -114,6 +158,7 @@ check_valid_header() { local file_lines=() # Read file line by line, handling newlines properly while IFS= read -r line; do + if [[ "$f" == *.md ]]; then line="${line%$'\r'}"; fi file_lines+=("$line") done < "$f" @@ -125,6 +170,15 @@ check_valid_header() { i=1 fi + if [[ "$f" == *.md ]]; then + local metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" + if [[ "$metadata_start" -gt 0 ]]; then + if [[ "$metadata_end" -eq 0 ]]; then return 1; fi + i=$metadata_end + fi + fi + # Extract the lines from expected_header_block local expected_lines=() # Use process substitution with IFS=$'\n' to split multi-line string into array elements @@ -140,7 +194,19 @@ check_valid_header() { # Compare line by line if [[ "${file_lines[i]:-}" == "${expected_lines[0]}" && "${file_lines[i+1]:-}" == "${expected_lines[1]}" ]]; then - return 0 # Exact header found + if [[ "$f" == *.md ]]; then + # Reject a conflicting declaration in the same bounded header region. + awk -v first="$((i + 3))" -v last="$((i + 15))" \ + -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + NR < first { next } + NR > last { exit } + /^[[:space:]]*$/ { next } + $0 ~ license_re || $0 ~ copyright_re { duplicate = 1; exit } + { exit } + END { exit duplicate ? 1 : 0 } + ' "$f" || return 1 + fi + return 0 # Exact unique header found fi return 1 } @@ -167,22 +233,45 @@ strip_existing_headers() { local temp_file temp_file=$(mktemp) + if [[ "$f" == *.md ]]; then + local metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" + awk -v metadata_start="$metadata_start" -v metadata_end="$metadata_end" \ + -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + BEGIN { header_active = 1 } + { + # The bounds helper admits only header comments and whitespace here. + if (metadata_start > 0 && NR <= metadata_end) { print; next } + if (header_active) { + if (NR > metadata_end + 15) header_active = 0 + else if ($0 ~ license_re || $0 ~ copyright_re) next + else if ($0 !~ /^[[:space:]]*$/) header_active = 0 + } + print + } + ' "$f" > "$temp_file" + cat "$temp_file" > "$f" + rm "$temp_file" + return + fi + # Use AWK to filter out lines in the first 15 lines that match SPDX/Copyright patterns. # This effectively removes "bad" headers or "wrong license" headers. # We preserve shebangs because they typically don't match the pattern. - awk ' + awk -v header_start=0 ' BEGIN { header_block_active = 1; line_num = 0 } { line_num++; + if (line_num <= header_start) { print; next; } if (header_block_active) { # Once we pass line 15, we are out of the header block. - if (line_num > 15) { header_block_active = 0; } + if (line_num > header_start + 15) { header_block_active = 0; } # If it is not a SPDX/Copyright line, and it is not a shebang/xml declaration, # then we are likely past the header block. # This condition is crucial for `in_header_block` to become 0. - if (line_num > 1 && $0 !~ /^#!/ && $0 !~ /^\<\?xml/ && $0 !~ /SPDX-License-Identifier/ && $0 !~ /James Ross .* FLYING/) { + if (line_num > header_start + 1 && $0 !~ /^#!/ && $0 !~ /^\<\?xml/ && $0 !~ /SPDX-License-Identifier/ && $0 !~ /James Ross .* FLYING/) { header_block_active = 0; } @@ -213,18 +302,37 @@ insert_header() { local first_line first_line=$(head -n 1 "$f" || true) + local header_cr="" + if [[ "$f" == *.md && "$first_line" == *$'\r' ]]; then + header_cr=$'\r' + header="${header//$'\n'/$'\r\n'}" + fi - # Logic to insert header AFTER shebang/xml declaration if present - if [[ "$first_line" =~ ^#! ]]; then + # Keep complete Markdown metadata ahead of the license in repair mode too. + local metadata_lines="" + if [[ "$f" == *.md ]]; then + local metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" + if [[ "$metadata_start" -eq 1 && "$metadata_end" -gt 0 ]]; then + metadata_lines="$metadata_end" + fi + fi + + # Logic to insert header AFTER metadata/shebang/xml declaration if present + if [[ -n "$metadata_lines" ]]; then + head -n "$metadata_lines" "$f" > "$temp_file" + printf '%s%s\n' "$header" "$header_cr" >> "$temp_file" + tail -n "+$((metadata_lines + 1))" "$f" >> "$temp_file" + elif [[ "$first_line" =~ ^#! ]]; then echo "$first_line" > "$temp_file" - echo "$header" >> "$temp_file" + printf '%s%s\n' "$header" "$header_cr" >> "$temp_file" tail -n +2 "$f" >> "$temp_file" elif [[ "$first_line" =~ ^\<\?xml ]]; then echo "$first_line" > "$temp_file" - echo "$header" >> "$temp_file" + printf '%s%s\n' "$header" "$header_cr" >> "$temp_file" tail -n +2 "$f" >> "$temp_file" else - echo "$header" > "$temp_file" + printf '%s%s\n' "$header" "$header_cr" > "$temp_file" cat "$f" >> "$temp_file" fi @@ -232,6 +340,22 @@ insert_header() { rm "$temp_file" } + +markdown_has_unclosed_header_attempt() { + local file="$1" metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$file")" + awk -v first="$((metadata_end + 1))" -v last="$((metadata_end + 15))" \ + -v license_re="$MD_LICENSE_COMMENT_PATTERN" -v copyright_re="$MD_COPYRIGHT_COMMENT_PATTERN" ' + NR < first { next } + NR > last { exit } + /^[[:space:]]*$/ { next } + /^/ { malformed = 1; exit } + $0 ~ license_re || $0 ~ copyright_re { next } + { exit } + END { exit malformed ? 0 : 1 } + ' "$file" +} + process_file() { local f="$1" if should_skip "$f"; then return; fi @@ -243,6 +367,21 @@ process_file() { local expected_header_block expected_header_block=$(get_header_content "$f" "$style") + if [[ "$f" == *.md ]]; then + local metadata_start metadata_end + read -r metadata_start metadata_end <<< "$(markdown_metadata_bounds "$f")" + if [[ "$metadata_start" -gt 0 && "$metadata_end" -eq 0 ]]; then + echo "[FAIL] Cannot process Markdown with unclosed frontmatter: $f" + FAILED_COUNT=$((FAILED_COUNT + 1)) + return + fi + if markdown_has_unclosed_header_attempt "$f"; then + echo "[FAIL] Cannot process Markdown with an unclosed license header: $f" + FAILED_COUNT=$((FAILED_COUNT + 1)) + return + fi + fi + if check_valid_header "$f" "$expected_header_block"; then return 0 # Header is already perfect, nothing to do. fi @@ -297,6 +436,10 @@ if [[ "$CHECK_MODE" -eq 1 ]]; then exit 1 fi else + if [[ "$FAILED_COUNT" -gt 0 ]]; then + echo "SPDX Repair Failed: $FAILED_COUNT files could not be repaired." + exit 1 + fi if [[ "$MODIFIED_COUNT" -gt 0 ]]; then echo "-------------------------------------------------------" echo "Repaired SPDX headers in $MODIFIED_COUNT files." diff --git a/scripts/tests/spdx_frontmatter_test.sh b/scripts/tests/spdx_frontmatter_test.sh new file mode 100755 index 00000000..89c7e967 --- /dev/null +++ b/scripts/tests/spdx_frontmatter_test.sh @@ -0,0 +1,492 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# © James Ross Ω FLYING•ROBOTS +set -euo pipefail +root=$(git rev-parse --show-toplevel) +cd "$root" +checker=${1:-scripts/ensure_spdx.sh} +# Every selected regression first runs the required valid/missing/wrong/header +# baseline. Selectors narrow the added regression, not this preflight. A RED +# assertion is evidence only after the baseline passes. +case "${SPDX_CASE:-all}" in + all|unclosed|header_first|displaced|prose|thematic|malformed|indented_keys|indented_comments|quoted_keys|spaced_keys|duplicate_headers|explicit_keys|unclosed_license|quoted_type|crlf|delimiterless|empty_id|type_comment|sequence_root|comment_id|flow_root|fenced_task|copyright_attempt|empty_yaml_id|spaced_delimiters|legacy_body) ;; + *) echo 'unknown SPDX regression case' >&2; exit 2 ;; +esac +work=$(mktemp -d "${TMPDIR:-/tmp}/echo-spdx-test.XXXXXX") +trap 'rm -rf "$work"' EXIT +cat > "$work/valid.md" <<'DOC' +--- +id: S01 +type: Feature +--- + + +# Task +DOC +bash "$checker" --check "$work/valid.md" +cat > "$work/missing.md" <<'DOC' +--- +id: S01 +--- +# Task +DOC +if bash "$checker" --check "$work/missing.md"; then + echo 'missing license passed' >&2 + exit 1 +fi +if bash "$checker" "$work/missing.md"; then + echo 'repair must report changed files' >&2 + exit 1 +else + test "$?" = 1 +fi +bash "$checker" --check "$work/missing.md" +test "$(head -n 1 "$work/missing.md")" = '---' +test "$(sed -n '2p' "$work/missing.md")" = 'id: S01' +sed 's/Apache-2.0 OR LicenseRef-MIND-UCAL-1.0/MIT/' "$work/valid.md" > "$work/wrong.md" +if bash "$checker" "$work/wrong.md"; then + echo 'repair must report changed files' >&2 + exit 1 +else + test "$?" = 1 +fi +cmp "$work/valid.md" "$work/wrong.md" +cat > "$work/unclosed.md" <<'DOC' +--- +id: S01 + + +DOC +if bash "$checker" --check "$work/unclosed.md"; then + echo 'unclosed frontmatter passed' >&2 + exit 1 +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == unclosed ]]; then +cp "$work/unclosed.md" "$work/unclosed.original" +if bash "$checker" "$work/unclosed.md"; then + echo 'unclosed repair passed' >&2 + exit 1 +else + test "$?" = 1 +fi +cmp "$work/unclosed.original" "$work/unclosed.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == header_first ]]; then +cat > "$work/header-first.md" <<'DOC' + + + +--- +id: S01 +type: Feature +--- +# Task +DOC +cp "$work/header-first.md" "$work/header-first.original" +bash "$checker" --check "$work/header-first.md" +bash "$checker" "$work/header-first.md" +cmp "$work/header-first.original" "$work/header-first.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == displaced ]]; then +cat > "$work/displaced.md" <<'DOC' +--- +id: S01 +--- + + + + +# Task +DOC +if bash "$checker" "$work/displaced.md"; then + echo 'displaced license must report a change' >&2 + exit 1 +else + test "$?" = 1 +fi +bash "$checker" --check "$work/displaced.md" +test "$(grep -c '^' + printf '%s\n' '' + printf '\n---\n# Body heading\nA normal paragraph.\n' + if [[ "$separator_count" = 2 ]]; then printf '\n---\nMore body.\n'; fi + } > "$work/thematic-$separator_count.md" + cp "$work/thematic-$separator_count.md" "$work/thematic-$separator_count.original" + bash "$checker" --check "$work/thematic-$separator_count.md" + bash "$checker" "$work/thematic-$separator_count.md" + cmp "$work/thematic-$separator_count.original" "$work/thematic-$separator_count.md" +done +printf '%s\n' '---' '# Body heading' 'A normal paragraph.' > "$work/unlicensed-break.md" +cp "$work/unlicensed-break.md" "$work/unlicensed-break.original" +if bash "$checker" --check "$work/unlicensed-break.md"; then exit 1; fi +if bash "$checker" "$work/unlicensed-break.md"; then exit 1; else test "$?" = 1; fi +tail -n +3 "$work/unlicensed-break.md" > "$work/unlicensed-break.body" +cmp "$work/unlicensed-break.original" "$work/unlicensed-break.body" +bash "$checker" --check "$work/unlicensed-break.md" +cat > "$work/note-body.md" <<'DOC' + + + +--- +Note: this section explains metadata. + +A normal body paragraph. +--- +DOC +cp "$work/note-body.md" "$work/note-body.original" +bash "$checker" --check "$work/note-body.md" +bash "$checker" "$work/note-body.md" +cmp "$work/note-body.original" "$work/note-body.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == malformed ]]; then +cat > "$work/malformed-comment.md" <<'DOC' +--- +id: S01 +--- + + +# Task +DOC +if bash "$checker" "$work/malformed-comment.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/malformed-comment.md" +test "$(grep -c 'SPDX-License-Identifier' "$work/malformed-comment.md")" = 1 +grep -qx '# Task' "$work/malformed-comment.md" +cat > "$work/equals-comment.md" <<'DOC' +--- +id: S01 +--- + + +# Task +DOC +if bash "$checker" "$work/equals-comment.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/equals-comment.md" +test "$(grep -c 'SPDX-License-Identifier' "$work/equals-comment.md")" = 1 +grep -qx '# Task' "$work/equals-comment.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == indented_keys ]]; then +cat > "$work/indented-keys.md" <<'DOC' + + + +--- +Example: a task-card code sample follows. + + id: S01 + type: Feature +--- +DOC +cp "$work/indented-keys.md" "$work/indented-keys.original" +bash "$checker" --check "$work/indented-keys.md" +bash "$checker" "$work/indented-keys.md" +cmp "$work/indented-keys.original" "$work/indented-keys.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == indented_comments ]]; then +cat > "$work/indented-comments.md" <<'DOC' +--- +id: S01 +--- + + + + +Body. +DOC +tail -n +4 "$work/indented-comments.md" > "$work/indented-comments.expected" +if bash "$checker" "$work/indented-comments.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/indented-comments.md" +tail -n +6 "$work/indented-comments.md" > "$work/indented-comments.actual" +cmp "$work/indented-comments.expected" "$work/indented-comments.actual" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == quoted_keys ]]; then +for key in '"title"' "'title'"; do +printf -- '---\n%s: Example\n---\n# Body\n' "$key" > "$work/quoted-keys.md" +head -n 3 "$work/quoted-keys.md" > "$work/quoted-keys.metadata" +if bash "$checker" "$work/quoted-keys.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/quoted-keys.md" +cp "$work/quoted-keys.md" "$work/quoted-keys.once" +bash "$checker" "$work/quoted-keys.md" +cmp "$work/quoted-keys.once" "$work/quoted-keys.md" +head -n 3 "$work/quoted-keys.md" > "$work/quoted-keys.actual" +cmp "$work/quoted-keys.metadata" "$work/quoted-keys.actual" +done +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == spaced_keys ]]; then +cat > "$work/spaced-keys.md" <<'DOC' +--- +title : Example +--- +# Body +DOC +head -n 3 "$work/spaced-keys.md" > "$work/spaced-keys.expected" +if bash "$checker" "$work/spaced-keys.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/spaced-keys.md" +head -n 3 "$work/spaced-keys.md" > "$work/spaced-keys.actual" +cmp "$work/spaced-keys.expected" "$work/spaced-keys.actual" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == duplicate_headers ]]; then +cat > "$work/duplicate-header.md" <<'DOC' +--- +id: S01 +--- + + + +# Body +DOC +if bash "$checker" --check "$work/duplicate-header.md"; then exit 1; else test "$?" = 1; fi +if bash "$checker" "$work/duplicate-header.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/duplicate-header.md" +test "$(grep -c SPDX-License-Identifier "$work/duplicate-header.md")" = 1 +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == explicit_keys ]]; then +cat > "$work/explicit-keys.md" <<'DOC' +--- +? title +: Example +--- +# Body +DOC +head -n 4 "$work/explicit-keys.md" > "$work/explicit-keys.expected" +if bash "$checker" "$work/explicit-keys.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/explicit-keys.md" +head -n 4 "$work/explicit-keys.md" > "$work/explicit-keys.actual" +cmp "$work/explicit-keys.expected" "$work/explicit-keys.actual" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == unclosed_license ]]; then +cat > "$work/unclosed-license.md" <<'DOC' +--- +id: S01 +--- + + +--- +id: S01 +type: "Feature" +--- +# Body +DOC +cp "$work/quoted-type.md" "$work/quoted-type.original" +bash "$checker" --check "$work/quoted-type.md" +bash "$checker" "$work/quoted-type.md" +cmp "$work/quoted-type.original" "$work/quoted-type.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == crlf ]]; then +printf -- '---\r\nid: S01\r\n---\r\n# Body\r\n' > "$work/crlf.md" +head -n 3 "$work/crlf.md" > "$work/crlf.metadata" +tail -n +4 "$work/crlf.md" > "$work/crlf.body" +if bash "$checker" "$work/crlf.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/crlf.md" +head -n 3 "$work/crlf.md" > "$work/crlf.actual-metadata" +tail -n +6 "$work/crlf.md" > "$work/crlf.actual-body" +cmp "$work/crlf.metadata" "$work/crlf.actual-metadata" +cmp "$work/crlf.body" "$work/crlf.actual-body" +python3 - "$work/crlf.md" <<'CHECK' +from pathlib import Path +import sys +raw=Path(sys.argv[1]).read_bytes() +assert b"\n" not in raw.replace(b"\r\n", b"") +CHECK +cp "$work/crlf.md" "$work/crlf.once" +bash "$checker" "$work/crlf.md" +cmp "$work/crlf.once" "$work/crlf.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == delimiterless ]]; then +cat > "$work/delimiterless.md" <<'DOC' +--- +id: S01 +--- + + +# Body +DOC +if bash "$checker" "$work/delimiterless.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/delimiterless.md" +test "$(grep -c SPDX-License-Identifier "$work/delimiterless.md")" = 1 +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == empty_id ]]; then +cat > "$work/empty-id.md" <<'DOC' + + +--- +id: "" +type: Feature +--- +# Body +DOC +cp "$work/empty-id.md" "$work/empty-id.original" +bash "$checker" --check "$work/empty-id.md" +bash "$checker" "$work/empty-id.md" +cmp "$work/empty-id.original" "$work/empty-id.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == type_comment ]]; then +cat > "$work/type-comment.md" <<'DOC' + + +--- +id: S01 +type: Feature # task category +--- +# Body +DOC +cp "$work/type-comment.md" "$work/type-comment.original" +bash "$checker" --check "$work/type-comment.md" +bash "$checker" "$work/type-comment.md" +cmp "$work/type-comment.original" "$work/type-comment.md" +fi + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == sequence_root ]]; then +for sequence_item in '- item' ' - item' '-'; do + printf '%s\n' '---' "$sequence_item" > "$work/sequence-root.md" + cp "$work/sequence-root.md" "$work/sequence-root.original" + if bash "$checker" --check "$work/sequence-root.md"; then exit 1; fi + if bash "$checker" "$work/sequence-root.md"; then exit 1; else test "$?" = 1; fi + cmp "$work/sequence-root.original" "$work/sequence-root.md" +done +fi + + + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == comment_id ]]; then +cat > "$work/comment-id.md" <<'DOC' + + + +--- +id: # deliberately absent +type: Feature +--- +A preserved body paragraph. +DOC +cp "$work/comment-id.md" "$work/comment-id.original" +bash "$checker" --check "$work/comment-id.md" +bash "$checker" "$work/comment-id.md" +cmp "$work/comment-id.original" "$work/comment-id.md" +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == flow_root ]]; then +for flow_root in '[one, two]' '{key: value}'; do + printf '%s\n' '---' "$flow_root" > "$work/flow-root.md" + cp "$work/flow-root.md" "$work/flow-root.original" + if bash "$checker" --check "$work/flow-root.md"; then exit 1; fi + if bash "$checker" "$work/flow-root.md"; then exit 1; else test "$?" = 1; fi + cmp "$work/flow-root.original" "$work/flow-root.md" +done +fi + + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == fenced_task ]]; then +for fence in '```' '````' '~~~' '~~~~'; do + { + printf '%s\n' '' '' '' '---' 'Note: task card example.' + printf '%syaml\n' "$fence" + printf '%s\n' 'id: S01' 'type: Feature' "$fence" '---' 'Preserved body.' + } > "$work/fenced-task.md" + cp "$work/fenced-task.md" "$work/fenced-task.original" + bash "$checker" --check "$work/fenced-task.md" + bash "$checker" "$work/fenced-task.md" + cmp "$work/fenced-task.original" "$work/fenced-task.md" +done +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == copyright_attempt ]]; then +printf '%s\n' '' '' '# Preserved body' > "$work/copyright-attempt.md" +if bash "$checker" "$work/copyright-attempt.md"; then exit 1; else test "$?" = 1; fi +bash "$checker" --check "$work/copyright-attempt.md" +test "$(grep -c '© James Ross' "$work/copyright-attempt.md")" = 1 +grep -qx '# Preserved body' "$work/copyright-attempt.md" +fi + + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == empty_yaml_id ]]; then +for id_value in NULL Null '[]' '{}'; do + printf '%s\n' '' '' '' '---' "id: $id_value" 'type: Feature' '---' 'Preserved body.' > "$work/empty-yaml-id.md" + cp "$work/empty-yaml-id.md" "$work/empty-yaml-id.original" + bash "$checker" --check "$work/empty-yaml-id.md" + bash "$checker" "$work/empty-yaml-id.md" + cmp "$work/empty-yaml-id.original" "$work/empty-yaml-id.md" +done +fi +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == spaced_delimiters ]]; then +for shape in opening closing both; do + opening='---';closing='---' + if [[ "$shape" != closing ]]; then opening='--- '; fi + if [[ "$shape" != opening ]]; then closing=$'---\t'; fi + printf '%s\n' "$opening" 'id: S01' "$closing" '# Body' > "$work/spaced-delimiters.md" + head -n 3 "$work/spaced-delimiters.md" > "$work/spaced-delimiters.original" + if bash "$checker" "$work/spaced-delimiters.md"; then exit 1; else test "$?" = 1; fi + bash "$checker" --check "$work/spaced-delimiters.md" + head -n 3 "$work/spaced-delimiters.md" > "$work/spaced-delimiters.actual" + cmp "$work/spaced-delimiters.original" "$work/spaced-delimiters.actual" +done +fi + + +# Closed companion controls preserve the deliberate unclosed refusal witnesses. +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == sequence_root || "${SPDX_CASE:-all}" == flow_root ]]; then +for root_value in '- item' ' - item' '-' '[one, two]' '{key: value}'; do + printf '%s\n' '---' "$root_value" '---' > "$work/closed-root.md" + cp "$work/closed-root.md" "$work/closed-root.original" + if bash "$checker" "$work/closed-root.md"; then exit 1; else test "$?" = 1; fi + bash "$checker" --check "$work/closed-root.md" + head -n 3 "$work/closed-root.md" > "$work/closed-root.actual" + cmp "$work/closed-root.original" "$work/closed-root.actual" + test "$(sed -n '4p' "$work/closed-root.md")" = '' + test "$(sed -n '5p' "$work/closed-root.md")" = '' +done +fi + + +if [[ "${SPDX_CASE:-all}" == all || "${SPDX_CASE:-all}" == legacy_body ]]; then +cat > "$work/legacy-body.md" <<'DOC' + + + +--- +Note: a body section can contain task-like fields. +id: S01 +type: Feature +--- +Preserved body. +DOC +cp "$work/legacy-body.md" "$work/legacy-body.original" +bash "$checker" --check "$work/legacy-body.md" +bash "$checker" "$work/legacy-body.md" +cmp "$work/legacy-body.original" "$work/legacy-body.md" +fi +printf '%s\n' 'PASS: metadata placement, unclosed repair refusal, displaced license removal, and prose preservation' diff --git a/tasks/K01.md b/tasks/K01.md new file mode 100644 index 00000000..df24a622 --- /dev/null +++ b/tasks/K01.md @@ -0,0 +1,77 @@ +--- +id: K01 +type: Feature +title: Prove the Echo and Keep content identity bridge +issue: https://github.com/flyingrobots/echo/issues/759 +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [] +status: complete +pr: https://github.com/flyingrobots/echo/pull/768 +integration_commit: bb20c57345374f476fa938789294a0890341eadd +--- + + + +# Feature + +```text +Task K01 completed in PR #768 at bb20c57345374f476fa938789294a0890341eadd. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Prove the Echo and Keep content identity bridge. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. + +## 3. Proposed Solution + +Read both exact identity laws. Compute both identities from one bounded source stream. Reconstruct through Keep. Verify the Echo hash and exact length again. Keep the binding private and versioned. + +Echo hashes raw content bytes with BLAKE3. Keep hashes its versioned domain, the content, and the exact length. Verify each identity against its own golden vectors. Do not require the two digests to equal each other. Reconstruct from Keep, then verify the Echo identity and length again. + +Use an isolated experimental adapter package with an explicit Rust 1.96 requirement. Keep the existing echo-cas Rust 1.90 package posture and default dependency graph intact. + +## 4. Prerequisites + +- [x] Recheck exact Echo and Keep identity source and supported toolchains. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +A golden-vector suite proves same-source and reconstructed-source agreement. A wrong identity or length refuses. The old Echo CAS remains the default. + +## 7. Definition of Done + +Completed in [PR #768](https://github.com/flyingrobots/echo/pull/768), integrated at `bb20c57345374f476fa938789294a0890341eadd`. Six identity witnesses, relevant guarded Docker and hosted checks, and current-head Code Lawyer and agy review passed. The bridge remains isolated and supplies no storage or durability proposition. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Empty bytes, text, binary ramps, chunk boundaries, interrupted sources, substituted identities, and length mismatches. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K02.md b/tasks/K02.md new file mode 100644 index 00000000..88b70ca1 --- /dev/null +++ b/tasks/K02.md @@ -0,0 +1,73 @@ +--- +id: K02 +type: Feature +title: Add the Echo physical-content port and existing CAS adapters +issue: https://github.com/flyingrobots/echo/issues/760 +status: complete +pr: https://github.com/flyingrobots/echo/pull/769 +integration_commit: 2056c95fb891125cbfcc8405e438542e020b7f4c +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [] +--- + + + +# Feature + +```text +Task K02 completed in PR #769 at 2056c95fb891125cbfcc8405e438542e020b7f4c. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Add the Echo physical-content port and existing CAS adapters. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. + +## 3. Proposed Solution + +Add an Echo-owned fallible complete-object port. Implement MemoryTier and DiskTier adapters. Keep types inside backend adapters. Require private staging and atomic output promotion. Report unsupported evidence explicitly. + +## 4. Prerequisites + +- [x] Recheck the Echo physical-content boundary and existing CAS adapter contracts. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +Existing backends pass the same complete-object tests. No failed read exposes a partial object. Existing consumers retain compatible behavior. + +## 7. Definition of Done + +Completed in [PR #769](https://github.com/flyingrobots/echo/pull/769), integrated at `2056c95fb891125cbfcc8405e438542e020b7f4c`. The final `k02-fresh-fixture-gate` passed 36 CAS tests (17 unit, four disk, five physical-content and ten retention), with CAS source hashes matching the merged candidate. The initial gate had 34 before later regression tests were added. Relevant strict checks and current-head Code Lawyer and agy reviews passed. Existing consumers remain compatible; initial receipts establish no durability or authenticated absence. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Missing content, corrupted content, failed writes, failed promotion, resource refusal, and unchanged visible output after failure. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K03.md b/tasks/K03.md new file mode 100644 index 00000000..4c9f9fa5 --- /dev/null +++ b/tasks/K03.md @@ -0,0 +1,74 @@ +--- +id: K03 +type: Feature +title: Add an experimental Keep ReferenceStore adapter +issue: https://github.com/flyingrobots/echo/issues/761 +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [K01, K02] +status: complete +pr: https://github.com/flyingrobots/echo/pull/770 +integration_commit: fe8789263a26fbcb7c7554c2b48f9c33b812c7eb +--- + + + +# Feature + +```text +Task K03 completed in PR #770 at fe8789263a26fbcb7c7554c2b48f9c33b812c7eb. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Add an experimental Keep ReferenceStore adapter. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. + +## 3. Proposed Solution + +Pin the inspected Keep revision. Add an optional Echo adapter over Keep ReferenceStore. Reuse the port conformance suite. Keep the adapter disabled by default and label its receipts non-durable. Keep concrete backend causes and coordinates private on error paths, including formatting, downcast and source chains. + +## 4. Prerequisites + +- [x] Complete [K01](K01.md). Its contract supplies required behavior. +- [x] Complete [K02](K02.md). Its contract supplies required behavior. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. GitHub records #759 and #760 as blockers of #761. Recheck the accepted edges before execution. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +Both existing CAS and Keep pass one backend-neutral suite. The adapter changes no Echo or WSC identity. No ReferenceStore receipt claims restart durability. + +## 7. Definition of Done + +Completed in [PR #770](https://github.com/flyingrobots/echo/pull/770), integrated at `fe8789263a26fbcb7c7554c2b48f9c33b812c7eb`. Thirteen feature-enabled witnesses, six default identity witnesses, the 36-test CAS suite, relevant guarded Docker and hosted checks, and current-head Code Lawyer and agy review passed. The adapter remains disabled by default and volatile. Its error boundary keeps concrete Keep causes private. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Byte equality, identity substitution, missing chunks, corrupt layouts, deterministic layout selection, capacity limits, source interruption, and public error-coordinate isolation. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K04.md b/tasks/K04.md new file mode 100644 index 00000000..e1ca7d3f --- /dev/null +++ b/tasks/K04.md @@ -0,0 +1,77 @@ +--- +id: K04 +type: Feature +title: Add a pinned-generation durable Keep read adapter +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [K03] +execution_scope: conditional_follow_on +external_prerequisites: [admitted_guarded_storage] +--- + + + +# Feature + +```text +Execute task K04 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Add a pinned-generation durable Keep read adapter. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. + +## 3. Proposed Solution + +Verify current Keep DurableStore and DurableView contracts at the selected revision. This task is outside the current experimental-adapter execution scope. + +First prove Keep admission on the exact backing filesystem. The durable profile requires Linux ext4 and its stated directory, mount, and synchronization conditions. A Linux container over overlayfs does not establish admission. If guarded admitted storage is unavailable, record the blocker. Do not forge platform evidence or create an unguarded filesystem. + +Run required tests in Docker. Do not use host test fallbacks. Unsupported-platform checks require a permitted Docker or cross-target route that preserves the execution policy. + +Add an optional adapter for an admitted immutable generation. Retain its evidence during the read. Reconstruct into bounded private staging and validate the full receipt. + +## 4. Prerequisites + +- [x] Complete [K03](K03.md). Its contract supplies required behavior. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +The adapter names and retains the exact view. Every success binds the requested content and length. Unsupported platforms refuse. This read adapter does not claim durable ingestion. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Generation changes, reader fencing, unsupported platforms, stale retention roots, partial output, corrupt retained closure, cancellation, and allocation limits. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K05.md b/tasks/K05.md new file mode 100644 index 00000000..29fb2563 --- /dev/null +++ b/tasks/K05.md @@ -0,0 +1,78 @@ +--- +id: K05 +type: Feature +title: Prove durable Echo and Keep publication reconciliation +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [K04] +execution_scope: conditional_follow_on +readiness: blocked_external +external_prerequisites: [keep_durable_ingestion, durable_operation_lookup, non_expiring_retention_anchor, admitted_guarded_storage] +--- + + + +# Feature + +```text +Execute task K05 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Prove durable Echo and Keep publication reconciliation. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. + +## 3. Proposed Solution + +First verify that Keep exposes durable ingestion, publication, retention anchors, and operation lookup. If any capability is absent, record the blocker on #722. After those prerequisites exist, retain one operation identity. Commit the Echo request before the physical effect. Publish Keep content under a non-expiring anchor. Settle Echo before dependent work resumes. Finalize retention before anchor release. + +## 4. Prerequisites + +- [ ] Keep exposes and verifies durable ingestion for the selected profile. +- [ ] Keep exposes durable operation lookup and a non-expiring reconciliation anchor. +- [ ] The shared guard accounts for the admitted backing store and every output. + +These external requirements are unresolved at Keep `3165890e`. No external issue ID is claimed. Issue #722 must record the accepted prerequisites and actual tracker links before this task can execute. + +- [ ] Complete [K04](K04.md). Its contract supplies required behavior. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +The bounded crash matrix proves idempotent recovery. An authoritative Echo reference never names unavailable physical content. The default backend remains unchanged. Missing prerequisite capabilities block execution. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Process death before and after each request, publication, settlement, and retention boundary. Duplicate retries, unavailable Echo, corrupt evidence, and cancellation. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/K06.md b/tasks/K06.md new file mode 100644 index 00000000..d9f95ac5 --- /dev/null +++ b/tasks/K06.md @@ -0,0 +1,71 @@ +--- +id: K06 +type: Feature +title: Prepare the migration and production adoption decision +tracking_issue: https://github.com/flyingrobots/echo/issues/722 +echo_evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +keep_evidence_commit: 3165890e9291cfb5fe10e81a9d7cd151f3e59464 +depends_on: [K05] +execution_scope: conditional_follow_on +readiness: blocked_external +--- + + + +# Feature + +```text +Execute task K06 under Echo issue #722. Recheck both repositories and the canonical boundary. Confirm all prerequisites. Preserve unrelated work. Use bounded Docker evidence. Apply the smallest independently mergeable change. Keep causal authority in Echo. Do not freeze public types or persisted bindings without a separately reviewed decision. Obtain Code Lawyer and agy approval before an authorized merge. Stop at the task completion signal. +``` + +## 1. Background Context + +The user requests Keep as Echo's physical CAS backend. Issue #722 owns integration. The canonical contract accepts experimental conformance. It does not accept production replacement. + +Echo source: `crates/echo-cas/src/lib.rs`, `crates/echo-cas/src/disk.rs`, and `crates/warp-core/src/wsc/store.rs`. + +Keep source: `src/lib.rs`, `src/reference/`, and `src/adapters/durable/` at `3165890e9291cfb5fe10e81a9d7cd151f3e59464`. + +## 2. Problem Description + +Prepare the migration and production adoption decision. Echo has the merged physical-content port, MemoryTier/DiskTier adapters, identity bridge, and optional in-memory Keep backend. Durable integration remains conditional. + +## 3. Proposed Solution + +Run bounded shadow comparison, backfill, and rollback rehearsals. Retain both content identities and explicit backend provenance. Present the exact platform, durability, retention, and compatibility evidence. Record the accepted decision in the canonical physical-content document before any production cutover. + +## 4. Prerequisites + +- [ ] Complete [K05](K05.md). Its contract supplies required behavior. + +Recheck external capability prerequisites before execution. Record accepted edges on #722. These task edges are evidence-backed proposals, not recorded tracker dependencies. + +## 5. Scope + +In: The stated integration slice and its executable evidence. Use an optional backend where code is experimental. + +Out: Graph ontology changes, WSC wire-identity changes, application-specific objects, silent fallback, and removal of echo-cas. + +## 6. Acceptance Criteria + +The reviewer can assess the complete adoption evidence. The decision records the selected posture and unresolved limits. No production cutover occurs without an explicit accepted decision. + +## 7. Definition of Done + +The named contract and witness pass. Current documents state the exact supported posture. One coherent PR records the evidence and integration commit. + +## 8. Test Plan + +Golden: A focused adapter or identity conformance test proves the stated contract. Name its exact command before code changes. + +Edges: Partial migration, substituted binding, unavailable Keep, explicit migration fallback, rollback, and fallback removal. + +Known failure modes: A backend receipt can overstate identity, complete-object coverage, visibility, or durability. Each negative witness must prove refusal or operational failure. + +Fuzz and stress: Use deterministic bounded fixtures. Enforce the shared build, data, and log limits. Do not start an unbounded campaign. + +## 9. Stakeholders + +James Ross owns the adoption decision. Echo consumers need stable content identity and causal evidence. Keep consumers need exact physical bytes and honest receipts. + +Related: [Echo issue #722](https://github.com/flyingrobots/echo/issues/722) and [the physical-content boundary](../docs/architecture/echo-keep-physical-content-boundary.md). diff --git a/tasks/S01.md b/tasks/S01.md new file mode 100644 index 00000000..2cb946ca --- /dev/null +++ b/tasks/S01.md @@ -0,0 +1,78 @@ +--- +id: S01 +type: Feature +title: Describe the reachable-state boundary of WorldlineState::state_root +issue: https://github.com/flyingrobots/echo/issues/754 +evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +depends_on: [] +status: complete +pr: https://github.com/flyingrobots/echo/pull/762 +integration_commit: da929ca6093977e909af20ef918e2431ad9b338c +--- + + + +# Feature + +```text +Task S01 completed in PR #762 at da929ca6093977e909af20ef918e2431ad9b338c. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. +``` + +## 1. Background Context + +Feedback item 1 in `FEEDBACK-echo.md` describes this problem. Echo source at `a93e9d82e89455ed1fa0b63447c88de544b9da26` confirms the mechanism. + +Source: https://github.com/flyingrobots/echo/blob/a93e9d82e89455ed1fa0b63447c88de544b9da26/crates/warp-core/src/worldline_state.rs#L249 + +The original experiment files are absent. The reported timings and counts are source claims, not reproduced measurements. + +## 2. Problem Description + +Historical defect at the audited revision: + +The API calls state_root() a full-state hash. The implementation excludes nodes that the root cannot reach. Create-if-absent produces such nodes. + +Current result: The API and owning documents state the reachable-state boundary. Detached writes remain bound by patch and commit identities; hash bytes are unchanged. + +## 3. Proposed Solution + +The merged repair followed this original scope: + +Correct the API contract and the operation documentation. Preserve the version-1 hash law. Explain that patch and commit identities retain detached writes. + +## 4. Prerequisites + +- [x] Use the shared guarded Docker worker for executable checks. +- [x] Record RED evidence before the code repair, or before/after evidence for a documentation repair. + +No other PR must merge first. Execution order does not establish a dependency. + +## 5. Scope + +In: This confirmed defect, its smallest witness, current owning documentation, and CHANGELOG when behavior changes. + +Out: Application-specific APIs, provider schema changes, new hash versions, and unrelated review findings. + +## 6. Acceptance Criteria + +The API states the reachable-state boundary. `docs/architecture/application-contract-hosting.md` and `docs/topics/WAL.md` state that a detached cell can leave the state root unchanged. A witness binds the write through its patch and commit. + +## 7. Definition of Done + +Completed in [PR #762](https://github.com/flyingrobots/echo/pull/762), integrated at `da929ca6093977e909af20ef918e2431ad9b338c`. The named witness, relevant checks and current-head Code Lawyer and agy review passed. GitHub owns any reopened or follow-on work. + +## 8. Test Plan + +Golden: Inspect the old and new API text. Run a focused warp-core test that creates two detached cells, compares roots, and checks distinct retained patches and commits. + +Edges: Reachable nodes, detached nodes, descended instances, and duplicate no-mutation evidence. + +Known failure modes at the audited revision: The API calls state_root() a full-state hash. The implementation excludes nodes that the root cannot reach. Create-if-absent produces such nodes. + +Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. + +## 9. Stakeholders + +James Ross owns Echo. Echo host authors need correct evidence and clear errors. Application authors need the documented runner contract. + +Related: #699. The existing target-value digest remains necessary. diff --git a/tasks/S02.md b/tasks/S02.md new file mode 100644 index 00000000..a192c2aa --- /dev/null +++ b/tasks/S02.md @@ -0,0 +1,78 @@ +--- +id: S02 +type: Feature +title: Run xtask run-edict-operation without a Git checkout or directory change +issue: https://github.com/flyingrobots/echo/issues/755 +evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +depends_on: [] +status: complete +pr: https://github.com/flyingrobots/echo/pull/765 +integration_commit: 6ef53c42db04ef16fae9e90e847203453a211af3 +--- + + + +# Feature + +```text +Task S02 completed in PR #765 at 6ef53c42db04ef16fae9e90e847203453a211af3. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. +``` + +## 1. Background Context + +Feedback item 5 in `FEEDBACK-echo.md` describes this problem. Echo source at `a93e9d82e89455ed1fa0b63447c88de544b9da26` confirms the mechanism. + +Source: https://github.com/flyingrobots/echo/blob/a93e9d82e89455ed1fa0b63447c88de544b9da26/xtask/src/main.rs#L454 + +The original experiment files are absent. The reported timings and counts are source claims, not reproduced measurements. + +## 2. Problem Description + +Historical defect at the audited revision: + +main() resolves a Git root and changes the current directory before it parses the command. The operation runner needs supplied artifact paths. It fails outside Git and resolves relative paths from a different directory inside Git. + +Current result: The operation runner parses first and uses caller-relative paths without Git-root discovery or a directory change. Repository maintenance commands retain their root behavior. + +## 3. Proposed Solution + +The merged repair followed this original scope: + +Parse the command first. Exempt run-edict-operation from both Git-root discovery and the directory change. Keep both steps for repository commands. + +## 4. Prerequisites + +- [x] Use the shared guarded Docker worker for executable checks. +- [x] Record RED evidence before the code repair, or before/after evidence for a documentation repair. + +No other PR must merge first. Execution order does not establish a dependency. + +## 5. Scope + +In: This confirmed defect, its smallest witness, current owning documentation, and CHANGELOG when behavior changes. + +Out: Application-specific APIs, provider schema changes, new hash versions, and unrelated review findings. + +## 6. Acceptance Criteria + +The runner succeeds outside Git with absolute paths. It resolves relative artifact paths from the caller directory. Existing repository commands retain their root policy. + +## 7. Definition of Done + +Completed in [PR #765](https://github.com/flyingrobots/echo/pull/765), integrated at `6ef53c42db04ef16fae9e90e847203453a211af3`. The named witness, relevant checks and current-head Code Lawyer and agy review passed. GitHub owns any reopened or follow-on work. + +## 8. Test Plan + +Golden: `cargo test -p xtask --test run_edict_operation`. Covered by `runner_accepts_absolute_artifacts_outside_git` and `runner_preserves_relative_paths_in_an_unrelated_nested_repository`. + +Edges: Absolute paths, relative paths, nested directories, malformed inputs, and repository commands. + +Known failure modes at the audited revision: main() resolves a Git root and changes the current directory before it parses the command. The operation runner needs supplied artifact paths. It fails outside Git and resolves relative paths from a different directory inside Git. + +Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. + +## 9. Stakeholders + +James Ross owns Echo. Echo host authors need correct evidence and clear errors. Application authors need the documented runner contract. + +Related: #531, which describes repository-root discovery for repository commands. diff --git a/tasks/S03.md b/tasks/S03.md new file mode 100644 index 00000000..6927db6b --- /dev/null +++ b/tasks/S03.md @@ -0,0 +1,78 @@ +--- +id: S03 +type: Feature +title: Report the typed Action obstruction when the operation runner cannot commit +issue: https://github.com/flyingrobots/echo/issues/756 +evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +depends_on: [] +status: complete +pr: https://github.com/flyingrobots/echo/pull/766 +integration_commit: 18b22e362e986f3e2509856433d040f33dc81bd2 +--- + + + +# Feature + +```text +Task S03 completed in PR #766 at 18b22e362e986f3e2509856433d040f33dc81bd2. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. +``` + +## 1. Background Context + +Feedback item 6 in `FEEDBACK-echo.md` describes this problem. Echo source at `a93e9d82e89455ed1fa0b63447c88de544b9da26` confirms the mechanism. + +Source: https://github.com/flyingrobots/echo/blob/a93e9d82e89455ed1fa0b63447c88de544b9da26/xtask/src/run_edict_operation.rs#L379 + +The original experiment files are absent. The reported timings and counts are source claims, not reproduced measurements. + +## 2. Problem Description + +Historical defect at the audited revision: + +The runner discards a non-committed Action outcome. It reports only that the scheduler did not publish a committed outcome. ResultProjectionInvalid therefore disappears from the caller error. + +Current result: Both Action failure paths report bounded typed outcome categories. Raw records, invocation data and child backtraces are omitted; retained obstruction encoding is unchanged. + +## 3. Proposed Solution + +The merged repair followed this original scope: + +Match the first Action outcome and the unexpected duplicate outcome explicitly. Use the same bounded formatter for both paths. Report an obstruction through its fieldless kind. Report footprint conflict with a fixed category and a bounded count. Report an absent outcome separately. Do not print full Debug records or invocation data. Keep retained obstruction encoding unchanged. + +## 4. Prerequisites + +- [x] Use the shared guarded Docker worker for executable checks. +- [x] Record RED evidence before the code repair, or before/after evidence for a documentation repair. + +No other PR must merge first. Execution order does not establish a dependency. + +## 5. Scope + +In: This confirmed defect, its smallest witness, current owning documentation, and CHANGELOG when behavior changes. + +Out: Application-specific APIs, provider schema changes, new hash versions, and unrelated review findings. + +## 6. Acceptance Criteria + +An output-bound refusal names ResultProjectionInvalid. Each outcome summary stays within 256 UTF-8 bytes and contains no invocation data. The runner emits no successful report. Existing durable obstruction and duplicate checks still pass. + +## 7. Definition of Done + +Completed in [PR #766](https://github.com/flyingrobots/echo/pull/766), integrated at `18b22e362e986f3e2509856433d040f33dc81bd2`. The named witness, relevant checks and current-head Code Lawyer and agy review passed. GitHub owns any reopened or follow-on work. + +## 8. Test Plan + +Golden: `cargo test -p xtask --test run_edict_operation`. Use a canonical input whose retained projection exceeds its declared output bound. + +Edges: Missing outcome, projection refusal, footprint conflict, committed outcome, and duplicate obstruction. Check the message length and absence of raw record fields. + +Known failure modes at the audited revision: The runner discards a non-committed Action outcome. It reports only that the scheduler did not publish a committed outcome. ResultProjectionInvalid therefore disappears from the caller error. + +Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. + +## 9. Stakeholders + +James Ross owns Echo. Echo host authors need correct evidence and clear errors. Application authors need the documented runner contract. + +Related: #696. Stable projection reason sub-codes are a separate schema proposal, not required for this repair. diff --git a/tasks/S04.md b/tasks/S04.md new file mode 100644 index 00000000..04181bdd --- /dev/null +++ b/tasks/S04.md @@ -0,0 +1,87 @@ +--- +id: S04 +type: Feature +title: Avoid repeated prefix replay and unbounded state retention during Action WAL recovery +issue: https://github.com/flyingrobots/echo/issues/757 +status: complete +pr: https://github.com/flyingrobots/echo/pull/767 +integration_commit: 7dde48b223ed105c13a1b70afb6a1a07d1dc308e +evidence_commit: a93e9d82e89455ed1fa0b63447c88de544b9da26 +depends_on: [] +--- + + + +# Feature + +```text +Task S04 completed in PR #767 at 7dde48b223ed105c13a1b70afb6a1a07d1dc308e. Recheck its GitHub issue, current main and retained acceptance evidence. When they agree, report verified completion and stop. Require a new executable claim before changes. For a reopened issue, read the canonical owners, name a narrow witness, use bounded Docker RED/GREEN evidence, preserve compatibility and unrelated work, update owning documents, and obtain current-head Code Lawyer and agy approval before an authorized merge. +``` + +## 1. Background Context + +Feedback item 2 in `FEEDBACK-echo.md` describes this problem. Echo source at `a93e9d82e89455ed1fa0b63447c88de544b9da26` confirms the mechanism. + +Source: https://github.com/flyingrobots/echo/blob/a93e9d82e89455ed1fa0b63447c88de544b9da26/crates/warp-core/src/trusted_runtime_host.rs#L4196 + +The original experiment files are absent. The reported timings and counts are source claims, not reproduced measurements. + +## 2. Problem Description + +Historical defect at the audited revision: + +Recovery caches a full WorldlineState for every distinct basis tick. Each cache miss replays from the initial boundary when no checkpoint exists. Consecutive bases repeat all earlier patches and retain growing graph and tick-history copies. + +Current result: Recovery verifies exact obligations through ordered replay cursors and two sweeps, preserving retained protocol order. It retains one cursor per needed worldline and one transient Tick state. Counters describe replay-patch work and logical retained data, with no linear elapsed-time or process-RSS claim. + +## 3. Proposed Solution + +The merged repair followed this original scope: + +Index verification obligations by exact coordinate without changing retained protocol order. Use bounded replay cursors and the existing replay validation primitives. Preserve all basis, patch, receipt, decision, conflict, and initial-boundary checks. + +1. Validate record identities and order under the existing laws. Index receipt and Action basis obligations by worldline and tick. Index composite Ticks separately by their parent coordinate. Store references and compact preparation evidence, not full states. +2. Visit basis obligations in ascending tick order within each worldline. Validate the initial boundary. Restore only an independently checked initial state or checkpoint at or before the first needed coordinate. Advance one cursor through retained patches. Verify the root and commit at every step. Validate all obligations at each selected basis. Retain exact preparation evidence for later Tick reconstruction. +3. After every basis obligation is checked, visit composite Tick parents in ascending order within each worldline. Reuse or restart the cursor through the same checked replay primitives. Preserve scheduler ordering, conflicts, member indices, obstructions, result identities, and patch commitments. This second sweep permits an Action basis on another worldline without premature Tick reconstruction. +4. Retain at most one replay cursor per needed worldline and one transient Tick simulation state. Discard the simulation after its checks. Publish no live runtime change before validation succeeds. Add separate counters for prefix initializations, replayed patches, peak cursor states, and peak retained graph/history data. + +One sweep is an acceptable alternative only after a witness proves that every preparation is available before its Tick validation. Never re-evaluate BasisChanged as a committed Action. Sorting verification references must not repair a malformed retained order. + +Bound replay-patch applications by two verified history sweeps per parent-state validation call. Count other activation work separately. This bound describes replay applications, not total CPU time. State hashing and Tick simulation still have graph-dependent cost. Preserve the legacy test helper's stated meaning or add a separate structured counter helper. Do not silently redefine its result. + +## 4. Prerequisites + +- [x] Use the shared guarded Docker worker for executable checks. +- [x] Record RED evidence before the code repair, or before/after evidence for a documentation repair. + +No other PR must merge first. Execution order does not establish a dependency. + +## 5. Scope + +In: This confirmed defect, its smallest witness, current owning documentation, and CHANGELOG when behavior changes. + +Out: Application-specific APIs, provider schema changes, new hash versions, and unrelated review findings. + +## 6. Acceptance Criteria + +A multi-tick recovery witness counts applied replay patches and retained replay states. Applied replay-patch work stays within two history sweeps. At most one cursor exists per needed worldline, plus one transient Tick simulation state. The witness bounds retained graph and tick-history data, not just object count. Poisoned bases and reordered or incomplete outcomes still refuse. + +## 7. Definition of Done + +Completed in [PR #767](https://github.com/flyingrobots/echo/pull/767), integrated at `7dde48b223ed105c13a1b70afb6a1a07d1dc308e`. The named witness, relevant checks and current-head Code Lawyer and agy review passed. GitHub owns any reopened or follow-on work. + +## 8. Test Plan + +Golden: `cargo test -p warp-core --features native_rule_bootstrap,trusted_runtime,host_test --test executable_operation_pipeline_tests`. Add deterministic work counters for consecutive ticks and mixed basis coordinates. + +Edges: Shared bases, delayed stale bases, non-monotonic retained basis order, cross-worldline bases, multiple heads, checkpoints, batches, conflicts, and corrupted retained evidence. Compare accepted and refused postures with the old validator. + +Known failure modes at the audited revision: Recovery caches a full WorldlineState for every distinct basis tick. Each cache miss replays from the initial boundary when no checkpoint exists. Consecutive bases repeat all earlier patches and retain growing graph and tick-history copies. + +Fuzz and stress: No unbounded campaign. Use deterministic fixtures within the shared resource limits. + +## 9. Stakeholders + +James Ross owns Echo. Echo host authors need correct evidence and clear errors. Application authors need the documented runner contract. + +Related: #751 concerns inverse-intent recovery and has separate scope. No dependency is established.