Skip to content

Host roll to the current release and store format 5, rehearsed on both production store copies - #201

Merged
TJKoury merged 34 commits into
mainfrom
issue-660-roll-script
Nov 17, 2026
Merged

TJKoury merged 34 commits into
mainfrom
issue-660-roll-script

Conversation

@TJKoury

@TJKoury TJKoury commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Roll a production node from store format 4 to format 5 in one detached script

deployment/scripts/roll-to-format5.sh runs the outage steps of a host roll as one script started with systemd-run, so a lost SSH session cannot leave a host down. It checks before stopping anything (unit, bundles, the staged binary's view of the peer ID, the running node's peer ID, the disk gate, the staged runtime's self-test with up to five tries), then stops the node, swaps the bundle folder by rename, backs up the store's side files, converts with store-migrate --to 5 under a hard budget, checks the format-5 switch, starts the new node and requires ready plus the same peer ID. Any failure after the stop puts the old bundle and the format-4 store back and starts the old node by itself; every step is a ledger line. deployment/scripts/roll-to-format5.md is the operator guide.

deployment/scripts/roll-to-format5-rehearsal/ is its end-to-end test: real systemd as PID 1 in a linux/amd64 container, no network, a copy-on-write copy of a store, a generated test identity, the host's own old binary as the old node and the published release archive (checked against the release's checksum file) as the new one. A case compares:

  • every record before and after, by content address and exact bytes, read through the node's own scan API (missing, changed or unexplained records fail; the node's own startup $ICN records and the refresh of its local profile are the only differences allowed);
  • every feed's count and every standard's count from the node's stats;
  • the format-4 file hashes at the moment the old node has stopped and again after a failed conversion has been undone, before the old node starts again;
  • after a forward roll, one clean stop and start of the new node: no quarantined or P4_E_CORRUPT journal line, and the record and feed comparisons again.

Rehearsed with v1.1.6 on copies of both production stores taken 2026-11-15:

Case host-01 copy host-02 copy
Forward: switched, new node ready with the same peer ID, every record present with equal bytes, every feed count equal passed (4,419,593 records, 8,959 feeds) passed (all records in 13 standards, 670 feeds)
One clean stop and start of the new node, then the same comparisons passed, no quarantine line passed, no quarantine line
Failed conversion: format-4 store byte-identical, old node back, every record present passed (one-minute budget) passed (budget hit during the conversion)
A second run after either outcome is refused without touching the node passed passed

Emulated linux/amd64 on an arm64 Mac: host-01's conversion took 131 minutes; host-02's did not finish within 180 minutes, so its forward case used the same release's linux/arm64 build for the new node (17 minutes) with the host's amd64 binary as the old one. These times describe emulation, not the hosts.

Refs DigitalArsenal/spacedatanetwork-stack#660

🤖 Generated with Claude Code

TJKoury and others added 30 commits November 16, 2026 08:48
Co-Authored-By: Codex <noreply@openai.com>
Preserve stopped side files and restore the retained source after failed
conversion or readiness checks. Supply isolated store and release inputs
to the systemd rehearsal and retain each result and recovery ledger.

Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Build the systemd image with its distribution packages and retain isolated
networking for release and store checks.

Co-Authored-By: Codex <noreply@openai.com>
A stable hostname allows the release to generate its isolated test identity.
Docker's default hexadecimal hostname is refused by the identity guard.

Co-Authored-By: Codex <noreply@openai.com>
The v1.1.0 arm64 bundle rejects both format-4 copies. An empty-store
control reaches readiness and creates format-5 marker bytes under fsql4.
Conversion and recovery remain unverified on these copies.

Co-Authored-By: Codex <noreply@openai.com>
Pair each store with its verified installed binary and preserve the old and
new bundle layouts. Isolate bootstrap traffic to loopback and supply fresh
test passwords and engine caches.

Co-Authored-By: Codex <noreply@openai.com>
Create a sparse ext4 loop filesystem before the old node starts so the
precheck observes insufficient real free space without a rollout override.

Co-Authored-By: Codex <noreply@openai.com>
Use the current systemd invocation so standalone old binaries do not need
an external wallet module for a separate identity command.

Co-Authored-By: Codex <noreply@openai.com>
Expose stats on the loopback rehearsal listener without credentials and
share only the compiled engine cache across fresh-copy cases.

Co-Authored-By: Codex <noreply@openai.com>
Prewarm engine artifacts with no provider services configured. Permit a
forward setup retry only before rollout, retaining previous attempt logs
and verifying the old bundle before reuse.

Co-Authored-By: Codex <noreply@openai.com>
Record readiness, running identity and aggregate equality separately, and
exercise repeat refusal after a completed recovery before reporting an
aggregate mismatch. Keep the original preservation assertion unchanged.

Co-Authored-By: Codex <noreply@openai.com>
Write independent byte-preservation and conversion observations before the
strict aggregate comparison. Retain every assertion and its failure.

Co-Authored-By: Codex <noreply@openai.com>
Record successful old-node baselines, intermittent runtime-guard refusal,
and observed post-switch recovery while retaining the failed preservation
gate and limits of the evidence.

Co-Authored-By: Codex <noreply@openai.com>
Compare rehearsal records by content address and exact bytes, and hash
format 4 before recovery restarts the original service. Reclaim completed
case stores while retaining separate record evidence and logs.

Co-Authored-By: Codex <noreply@openai.com>
Record each runtime attempt and retain conversion output. Rehearse automatic service restarts with the deployment restart policy.

Co-Authored-By: Codex <noreply@openai.com>
Stats counts sum source associations. Validate each scan against its own total and retain both measurements beside the exact byte comparison.

Co-Authored-By: Codex <noreply@openai.com>
Retain completed baseline indexes with the evidence and compare against temporary copies. The record completeness and byte checks are unchanged.

Co-Authored-By: Codex <noreply@openai.com>
Request 50000 records per page while retaining complete snapshot and byte checks.

Co-Authored-By: Codex <noreply@openai.com>
Verify identical baseline bytes for repeated content addresses and require every occurrence after rollout. Extra or missing occurrences fail preservation.

Co-Authored-By: Codex <noreply@openai.com>
Use native Python with native systemd support while the release binaries retain their target architecture. Record assertions and scan limits are unchanged.

Co-Authored-By: Codex <noreply@openai.com>
Save duration, sampled peak memory and switched size when the rollout exits so later read failures preserve the measurements.

Co-Authored-By: Codex <noreply@openai.com>
Compare the selected binary with the baseline digest after recovery and precheck refusal, and preserve the result beside readiness and identity evidence.

Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rehearsal reads every feed's count from the node's own stats before and
after the roll and requires each to be unchanged, apart from the node's own
startup connector records and a refreshed local profile. Case stores are
copy-on-write clones of the source copy, the driver runs on Python 3.9, and
each case container takes its CPU and memory limits from arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a forward roll, the rehearsal stops and starts the new node once,
fails on any quarantined or P4_E_CORRUPT journal line from the stop on,
and repeats the record-by-record and per-feed comparisons against the
format-4 baseline. Each node start since the roll may add one connector
record per connector.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Startup connector records raise the per-standard $ICN count by exactly the
persisted count while the per-feed rows can stay unchanged, because they
can land in the local lane. Per-feed $ICN rows may rise by at most that
count; per-standard counts must be equal apart from that $ICN rise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TJKoury and others added 4 commits November 16, 2026 22:27
A 50,000-record scan page of an emulated node under load took longer
than two minutes. No comparison changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TJKoury
TJKoury merged commit e8d8cc4 into main Nov 17, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant