Repository navigation
fix: preserve workload recovery identities and removal intents - #761
Merged
Merged
Conversation
The create recovery needs only the record gate gone. vm inspect returns creating placeholders, so a failed rm on a VM that is still there is still an error.
doDeployOneWorkload is the only caller and always reserves the ID first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A lost create reply could leave a process or VM on the node after Core discarded its name-only recovery intent. A committed metadata delete whose reply was lost could similarly discard the removal intent before calling the engine. Replacement recovery also lost the node needed to clean up the old engine workload once its metadata disappeared.
Reserve the engine ID before the existing create journal write, pass it through creation, and use it for rollback and replay. Containerd keeps name-as-ID; process and Cocoon keep their existing random ID format. Cocoon removal now works before its Eru metadata record exists; a failed
vm rmon a VM thatvm inspectstill reports stays an error.Commit removal intents only when deletion or restoration has settled. Read back an uncertain store deletion and restore metadata when needed. Record the node in the existing replacement intent so replay can finish old-engine cleanup even if both workload records are gone; retain the old workload when its metadata still exists and the replacement is absent.
Origin and recovery gaps
Validation
go test -race -count=1 -timeout 180s ./...: 667 top-level tests plus 418 subtests passed, 38 test packages; 28 packages have no tests.make lint,make fmt-check,make vet, andasl ./...passed with the repository's pinned tools.go test -raceon./cluster/calcium/...and./engine/..., lint, fmt-check and asl locally, and all seven CI checks.Cost and compatibility
Successful create/remove/replace paths add no network requests, journal writes, scans, or locks. ID generation happens once, earlier; existing WAL payloads gain the ID or node. Failed store deletion adds read-back and optional restoration.
Legacy WAL entries remain readable, but this change cannot identify already orphaned random-ID workloads or missing-node replacement records from older versions. No automatic scan or migration is added.
All seven CI checks now pass. The first pull-request test run failed in the unchanged
store/etcdv3.TestNodeStatusStream, whose fixed three-second cancellation can precede asynchronous lease-expiry delivery; the same commit's push test passed. One rerun of that failed job passed without code changes.