Repository navigation
Host roll to the current release and store format 5, rehearsed on both production store copies - #201
Merged
Merged
Conversation
Co-Authored-By: Codex <noreply@openai.com>
Preserve stopped side files and restore the retained source after failed conversion or readiness checks. Supply isolated store and release inputs to the systemd rehearsal and retain each result and recovery ledger. Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Build the systemd image with its distribution packages and retain isolated networking for release and store checks. Co-Authored-By: Codex <noreply@openai.com>
A stable hostname allows the release to generate its isolated test identity. Docker's default hexadecimal hostname is refused by the identity guard. Co-Authored-By: Codex <noreply@openai.com>
The v1.1.0 arm64 bundle rejects both format-4 copies. An empty-store control reaches readiness and creates format-5 marker bytes under fsql4. Conversion and recovery remain unverified on these copies. Co-Authored-By: Codex <noreply@openai.com>
Pair each store with its verified installed binary and preserve the old and new bundle layouts. Isolate bootstrap traffic to loopback and supply fresh test passwords and engine caches. Co-Authored-By: Codex <noreply@openai.com>
Create a sparse ext4 loop filesystem before the old node starts so the precheck observes insufficient real free space without a rollout override. Co-Authored-By: Codex <noreply@openai.com>
Use the current systemd invocation so standalone old binaries do not need an external wallet module for a separate identity command. Co-Authored-By: Codex <noreply@openai.com>
Expose stats on the loopback rehearsal listener without credentials and share only the compiled engine cache across fresh-copy cases. Co-Authored-By: Codex <noreply@openai.com>
Prewarm engine artifacts with no provider services configured. Permit a forward setup retry only before rollout, retaining previous attempt logs and verifying the old bundle before reuse. Co-Authored-By: Codex <noreply@openai.com>
Record readiness, running identity and aggregate equality separately, and exercise repeat refusal after a completed recovery before reporting an aggregate mismatch. Keep the original preservation assertion unchanged. Co-Authored-By: Codex <noreply@openai.com>
Write independent byte-preservation and conversion observations before the strict aggregate comparison. Retain every assertion and its failure. Co-Authored-By: Codex <noreply@openai.com>
Record successful old-node baselines, intermittent runtime-guard refusal, and observed post-switch recovery while retaining the failed preservation gate and limits of the evidence. Co-Authored-By: Codex <noreply@openai.com>
Compare rehearsal records by content address and exact bytes, and hash format 4 before recovery restarts the original service. Reclaim completed case stores while retaining separate record evidence and logs. Co-Authored-By: Codex <noreply@openai.com>
Record each runtime attempt and retain conversion output. Rehearse automatic service restarts with the deployment restart policy. Co-Authored-By: Codex <noreply@openai.com>
Stats counts sum source associations. Validate each scan against its own total and retain both measurements beside the exact byte comparison. Co-Authored-By: Codex <noreply@openai.com>
Retain completed baseline indexes with the evidence and compare against temporary copies. The record completeness and byte checks are unchanged. Co-Authored-By: Codex <noreply@openai.com>
Request 50000 records per page while retaining complete snapshot and byte checks. Co-Authored-By: Codex <noreply@openai.com>
Verify identical baseline bytes for repeated content addresses and require every occurrence after rollout. Extra or missing occurrences fail preservation. Co-Authored-By: Codex <noreply@openai.com>
Use native Python with native systemd support while the release binaries retain their target architecture. Record assertions and scan limits are unchanged. Co-Authored-By: Codex <noreply@openai.com>
Save duration, sampled peak memory and switched size when the rollout exits so later read failures preserve the measurements. Co-Authored-By: Codex <noreply@openai.com>
Compare the selected binary with the baseline digest after recovery and precheck refusal, and preserve the result beside readiness and identity evidence. Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rehearsal reads every feed's count from the node's own stats before and after the roll and requires each to be unchanged, apart from the node's own startup connector records and a refreshed local profile. Case stores are copy-on-write clones of the source copy, the driver runs on Python 3.9, and each case container takes its CPU and memory limits from arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a forward roll, the rehearsal stops and starts the new node once, fails on any quarantined or P4_E_CORRUPT journal line from the stop on, and repeats the record-by-record and per-feed comparisons against the format-4 baseline. Each node start since the roll may add one connector record per connector. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Startup connector records raise the per-standard $ICN count by exactly the persisted count while the per-feed rows can stay unchanged, because they can land in the local lane. Per-feed $ICN rows may rise by at most that count; per-standard counts must be equal apart from that $ICN rise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 50,000-record scan page of an emulated node under load took longer than two minutes. No comparison changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roll a production node from store format 4 to format 5 in one detached script
deployment/scripts/roll-to-format5.shruns the outage steps of a host roll as one script started withsystemd-run, so a lost SSH session cannot leave a host down. It checks before stopping anything (unit, bundles, the staged binary's view of the peer ID, the running node's peer ID, the disk gate, the staged runtime's self-test with up to five tries), then stops the node, swaps the bundle folder by rename, backs up the store's side files, converts withstore-migrate --to 5under a hard budget, checks the format-5 switch, starts the new node and requires ready plus the same peer ID. Any failure after the stop puts the old bundle and the format-4 store back and starts the old node by itself; every step is a ledger line.deployment/scripts/roll-to-format5.mdis the operator guide.deployment/scripts/roll-to-format5-rehearsal/is its end-to-end test: real systemd as PID 1 in a linux/amd64 container, no network, a copy-on-write copy of a store, a generated test identity, the host's own old binary as the old node and the published release archive (checked against the release's checksum file) as the new one. A case compares:$ICNrecords and the refresh of its local profile are the only differences allowed);quarantinedorP4_E_CORRUPTjournal line, and the record and feed comparisons again.Rehearsed with v1.1.6 on copies of both production stores taken 2026-11-15:
Emulated linux/amd64 on an arm64 Mac: host-01's conversion took 131 minutes; host-02's did not finish within 180 minutes, so its forward case used the same release's linux/arm64 build for the new node (17 minutes) with the host's amd64 binary as the old one. These times describe emulation, not the hosts.
Refs DigitalArsenal/spacedatanetwork-stack#660
🤖 Generated with Claude Code