Skip to content

fix: preserve workload recovery identities and removal intents - #761

Merged
CMGS merged 5 commits into
masterfrom
codex/preserve-workload-recovery-intents
Oct 5, 2026
Merged

CMGS merged 5 commits into
masterfrom
codex/preserve-workload-recovery-intents

Conversation

@CMGS

@CMGS CMGS commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

A lost create reply could leave a process or VM on the node after Core discarded its name-only recovery intent. A committed metadata delete whose reply was lost could similarly discard the removal intent before calling the engine. Replacement recovery also lost the node needed to clean up the old engine workload once its metadata disappeared.

Reserve the engine ID before the existing create journal write, pass it through creation, and use it for rollback and replay. Containerd keeps name-as-ID; process and Cocoon keep their existing random ID format. Cocoon removal now works before its Eru metadata record exists; a failed vm rm on a VM that vm inspect still reports stays an error.

Commit removal intents only when deletion or restoration has settled. Read back an uncertain store deletion and restore metadata when needed. Record the node in the existing replacement intent so replay can finish old-engine cleanup even if both workload records are gone; retain the old workload when its metadata still exists and the replacement is absent.

Origin and recovery gaps

Validation

  • Full go test -race -count=1 -timeout 180s ./...: 667 top-level tests plus 418 subtests passed, 38 test packages; 28 packages have no tests.
  • Linux and Darwin: make lint, make fmt-check, make vet, and asl ./... passed with the repository's pinned tools.
  • Regression coverage includes reserved-ID journaling/replay, actual Redis-store deletion followed by injected reply loss, failed read-back/restoration, replacement cleanup with/without a surviving new workload, and real shell fixtures for removing a VM before its Eru record exists.
  • The Redis tests were also run independently under the race detector after fixing their engine-cache fixture initialization.
  • After review the typed stale-create reconciliation and the engines' empty-ID fallbacks were cut. The final head passed go test -race on ./cluster/calcium/... and ./engine/..., lint, fmt-check and asl locally, and all seven CI checks.
  • These are isolated local tests; live SSH/KVM/systemd failure injection was not run.

Cost and compatibility

Successful create/remove/replace paths add no network requests, journal writes, scans, or locks. ID generation happens once, earlier; existing WAL payloads gain the ID or node. Failed store deletion adds read-back and optional restoration.

Legacy WAL entries remain readable, but this change cannot identify already orphaned random-ID workloads or missing-node replacement records from older versions. No automatic scan or migration is added.

All seven CI checks now pass. The first pull-request test run failed in the unchanged store/etcdv3.TestNodeStatusStream, whose fixed three-second cancellation can precede asynchronous lease-expiry delivery; the same commit's push test passed. One rerun of that failed job passed without code changes.

CMGS added 5 commits October 5, 2026 02:01
The create recovery needs only the record gate gone. vm inspect returns creating placeholders, so a failed rm on a VM that is still there is still an error.
doDeployOneWorkload is the only caller and always reserves the ID first.
@CMGS
CMGS marked this pull request as ready for review October 5, 2026 07:31
@CMGS
CMGS merged commit dc94d3d into master Oct 5, 2026
7 checks passed
@CMGS
CMGS deleted the codex/preserve-workload-recovery-intents branch October 5, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant