Repository navigation
Round 2026-09-25 residue: :hibernate bookkeeping across suspend, unsuspend, image edits and migration; a stale status is rewritten - #61
Merged
Conversation
…bernate tag first An image edit made while the set was suspended, or in the same apply as suspend:false, made the unsuspend pass record the main's VM for reclaim, clear its hibernate annotation and then delete the main for drift before vk-cocoon woke it. vk's DeletePod keeps the registry tag, so the recreated main (fresh from createMainAgent, or restore-marked by the Suspended-phase wake) met a :hibernate tag whose recorded base image differs from the new one, and vk-cocoon failed the create with image_conflict. The failed main matched the spec, so step 7 parked the set in Failed and the record and the tag stayed until someone deleted the tag by hand. A toolbox rebuilt for an image change before its wake completes hits the same refusal and burns its rebuild budget. Before the drift rebuild deletes a main or toolbox whose VM is still recorded and whose image changed, the tag is deleted and the VM leaves the record, so the new pod boots fresh. Other drift keeps the tag, because vk restores it into the new pod. A refused delete keeps the VM recorded, fails the pass and leaves the pod in place, as the woken-VM reclaim does; that reclaim now goes through the same helper. The check costs no registry call unless the drifted pod's VM is recorded.
…e :hibernate tag first vk-cocoon refuses a fork-from pod that is not marked for restore while its VM still has a :hibernate tag (source_conflict), and its DeletePod keeps that tag. The reclaim record only drops a tag once the pod wakes, so a sub-agent recreated while its VM was still recorded could never start: the operator rebuilt it four times and dead-lettered the slot. Two plain paths got there after an unsuspend: a sub-agent whose wake failed was rebuilt as terminal, and a scale-down deleted an extra slot before it woke and a later scale-up brought the slot back. Before the sub-agent creates of a pass, each missing slot whose VM is still recorded and that no CocoonHibernation restores has its tag deleted and leaves the record, so the new pod forks fresh from the main. A slot a CocoonHibernation restores keeps the tag and is marked for restore as before. A refused delete keeps the VM recorded and fails the pass before any create. The step costs nothing unless the record is non-empty and a slot is missing, and then only the CocoonHibernation list the creates already share.
… main runs a live VM A spec.nodeName edit that landed while the main was still waking (a plain unsuspend, the in-place wake of a pinned set, or a release-policy restore) sent the stale-tag branch of advanceMigration to delete the :hibernate tag the wake was reading: the main was off target and not quiesced by this controller, so the tag looked like a leftover. vk-cocoon's wake verifies the local snapshot against that tag, so the wake failed, the failed main still matched the spec, and step 7 parked the set in Failed before the migration could run again. The stale-tag branch now hands a main whose VM is not yet live back to the normal flow, which lets the wake finish and reclaims a recorded tag as usual. Once the main runs a live VM, a remaining tag is a true leftover: the branch drops it and the migration starts on the next pass. The guard runs before the CocoonHibernation list, so the waiting pass costs one registry probe, as before.
… the failed-state short-circuit vk-cocoon fails every update of a macOS pod that carries hibernate=true with HibernateUnsupported. A suspended macOS CocoonSet therefore left its main failed, and on unsuspend the failed-state short-circuit (step 7) ran before the un-suspend step, so the annotation that kept failing was never cleared: the generation stamp of the unsuspend was refused again and the set stayed Failed until someone deleted the main by hand. A Linux main whose hibernate vk-cocoon refused parks the same way. The failed-main path now runs the un-suspend first: it records and clears the hibernate annotation of every owned pod a prior suspend left, skipping CocoonHibernation-owned pods as step 10 does, so the pod's next update carries hibernate=false. vk-cocoon lifts a HibernateUnsupported failure on such an update while the VM is still live, which returns the main to Ready, and the existing RecoveredFromFailure path resumes the set. A failed main that never suspended costs one scan of the classified pods.
…nstead of restoring A release-policy suspend deletes the pods and keeps the :hibernate tag as the wake source. Editing spec.agent.image while the seat was released made the wake recreate the main restore-marked with the new image, and vk-cocoon refuses to restore a snapshot pushed from another image (image_conflict), so the main failed, matched the spec, and parked the set in Failed until someone deleted the tag by hand. A set pinned by spec.nodeName wakes through the migration step's recreate and hit the same refusal. The release now stashes the main's image in cocoonset.cocoonstack.io/hibernated-image next to the node hint, and the hint and the image clear together when the wake completes. When a released wake or the migration recreate finds the spec image differs from the stashed one, it deletes the tag, clears the release record and lets the normal flow boot the main fresh from the new image, the same rule an image edit applies to an owed main. A refused delete fails the pass and retries with backoff. A release stashed before this change carries no image and restores as before. The check reads one annotation on the wake pass only.
…pshot on unsuspend
A suspend hibernates every sub-agent and pushes its :hibernate tag; the
suspend path never recreates a missing sub-agent. When a sub-agent pod was
deleted while the set was suspended, the unsuspend recorded only the pods
it found, so the slot was later created unmarked with fork-from, and
vk-cocoon refused it while the tag existed (source_conflict): four
rebuilds, then dead-letter, and the suspended state was stranded.
The unsuspend pass, the one that clears a hibernate annotation, now probes
the registry for each desired sub-agent slot whose pod is missing and that
no CocoonHibernation owns, and records a slot whose tag exists as owed a
restore in the reclaim record ("restore", omitted when empty, so existing
records keep their bytes). Step 12 creates such a slot marked for restore
(re-probing the tag, failing closed), the owed-fork reclaim leaves its tag
alone, and the woken-VM reclaim drops it from both lists once it wakes. A
restore that keeps failing retries within the rebuild budget and keeps the
tag. The probe runs only in the unsuspend pass and only for missing slots;
a steady-state pass makes no registry call, which the steady-state
reconcile test now asserts.
…resh instead of restoring The restore record ignored the image check the main and toolbox rebuilds apply. A sub-agent deleted while the set was suspended, then an unsuspend that also edited spec.agent.image, created the slot restore-marked under the new image; vk-cocoon refuses a snapshot pushed from another image (image_conflict), and the owed-fork reclaim skips a slot owed a restore, so the slot burned its rebuild budget and dead-lettered with the tag kept. The slot is now recorded as owed a restore only while the main pod runs the spec image; otherwise it is owed only, so step 12 drops its tag and forks it fresh.
…ops the tag and boots fresh vk-cocoon records the pod's image on every :hibernate manifest it pushes and refuses to create any pod, restore-marked or not, over a tag recorded from another image (image_conflict). Three per-path patches covered part of that: the drift rebuild of a recorded main or toolbox, the hibernated-image stash of a released main, and the unsuspend guessing a missing sub-agent's snapshot image from the main pod. Other creates still met the refusal: a migration recreate after an image edit in the same apply as a nodeName edit, or during the quiesce, parked Failed at step 7; a main deleted while suspended plus an image edit timed out every 3 min and parked Failed on unsuspend; a toolbox recreated after an image edit dead-lettered; a sub-agent hibernated on the old image beside a new-image main was restore-marked by the guess. A failed clear after the stashed-image discard also left a stale image that a later migration compared against, dropping the migration's own snapshot. Every agent and toolbox create (the main, the released wake, the migration recreate, the sub-agents, the toolboxes) now reads the VM's :hibernate manifest first. When its recorded base image is non-empty and differs from the new pod's image, the tag is deleted, the VM leaves the reclaim record (vms and restore), and the pod is created fresh. The manifest is the only record, so nothing can go stale. A refused read or delete fails the pass and retries with backoff, and no pod is created over the tag meanwhile. The three patches are removed; a released wake that finds no snapshot left clears the node hint, as the stashed-image discard did. Per create this costs one manifest GET for a managed pod (the sub-agent GETs run inside the existing create fan-out), plus one DELETE and at most one CocoonSet patch on a conflict. A pass that creates no pod makes no registry call.
…ep, not the migration On a set pinned by spec.nodeName, the migration step's in-place wake (a main quiesced on its target) also caught a plain unsuspend: it cleared the main's hibernate annotation and marked the set Migrating, so applyUnsuspend never ran for the main. When every sub-agent pod had been deleted while suspended, no pod still carried hibernate afterwards, applyUnsuspend returned before its missing-slot probe, and each slot was created as a fork-from pod while its :hibernate tag existed: vk-cocoon refused it (source_conflict) and the slot dead-lettered. The in-place wake is the migration's only while the set is Migrating, which startMigration persists before it quiesces, so a re-target back still wakes in place there. Otherwise the migration step leaves the main to applyUnsuspend, as for an unpinned set: it records the reclaim, probes the missing slots and clears the annotation, and the woken-VM reclaim drops the tag. The re-target-back test fixture now carries the Migrating phase such a pass always has. No new registry call: the pass probed the tag before and still does.
…f looping the migration The failed-main path runs the un-suspend first so a main whose suspend hibernate vk-cocoon refused recovers on unsuspend. That also cleared the hibernate annotation startMigration sets to quiesce the main. A macOS main pinned to another node looped: startMigration set hibernate, vk-cocoon failed the pod with HibernateUnsupported, the failed path cleared the annotation, vk-cocoon lifted the failure, the main went Ready and the next pass started the migration again. A Linux main whose push kept failing redid a save and push every cycle. Before the un-suspend moved into the failed path, such a main parked Failed. The failed path now clears hibernate only when no migration is pending for the main, i.e. unless spec.nodeName is set and the main runs on another node; a pending migration's failed main parks Failed as before, until the pin changes. The off-target test the migration step spelled out twice is now one helper shared by the three sites. No new API or registry call.
… was suspended The unsuspend pass probed the missing sub-agent slots only. A toolbox pod deleted while the set was suspended was never recorded: it was recreated unmarked, vk-cocoon restored it from its :hibernate tag on its own, and since the VM was not in the reclaim record the tag was never dropped, so every later recreate of that toolbox rolled it back to the suspend-time snapshot. The same probe now covers the desired toolboxes whose pod is missing and that no CocoonHibernation owns; a toolbox whose tag exists is recorded as owed a reclaim (vms, not restore, since vk-cocoon restores an unmarked toolbox itself), and the woken-VM reclaim drops its tag once it wakes. The gating is unchanged: only the pass that clears a hibernate annotation probes, and only for missing pods, so a steady-state pass still makes no registry call. Cost: one tag probe per missing toolbox on that pass.
… suspended restores on unsuspend The unsuspend learned what it owed from the pods that still carried the hibernate annotation, and probed the registry for missing sub-agent slots and toolboxes only in a pass that cleared one. Three reachable passes clear none: an unsuspend that also edits spec.nodeName, where the migration takes the suspend's hibernate as its quiesce and moves the main first; a retain set whose pods are all gone or terminating when the unsuspend is observed, where the wake restores the main first; and a main missing at an unsuspend from Suspending or a timed-out suspend, which the probe never covered. A sub-agent deleted while suspended then forked over its :hibernate tag (vk-cocoon source_conflict, dead-letter), and a missing main or toolbox restored from its tag without being recorded, so the tag lingered and a later recreate rolled back to the suspend-time snapshot. A set deleted while suspended also orphaned the tag of a pod deleted meanwhile, since the teardown GC reads only Status.Agents, the delete-vm-names stash and the live pods, and VM names are deterministic, so a same-name successor set restores or dead-letters over it. The suspend path now records every managed VM of the owned pods in the hibernate-reclaim record, marked suspended, before it sets the first hibernate annotation. The first un-suspend pass that sees the marker sets the record generation, records every recorded VM whose pod is missing as owed a restore, clears the annotations and clears the marker last, whether or not a pod still carries hibernate; the registry probe is removed. Creating the main, a sub-agent or a toolbox marks a VM owed a restore for restore through the existing fail-closed tag probe, after the image guard, so the three creates share that rule in markRestoreFromIntent. The migration is unchanged: it may move the main on the suspend's snapshot, which keeps its own invariant (the old pod dies only once the tag exists and the pod carries hibernate), and the bookkeeping runs on the first pass it hands back, before any sub-agent or toolbox is created; deferring the migration would instead wake the main on the old node and hibernate it again. A pod that carries hibernate without a marked record (a set suspended by an earlier operator, an abandoned migration quiesce) is still recorded and cleared as before. The teardown GC also takes the record's VMs. A pass on a set that never suspended makes no new API call. The first suspend pass adds one CocoonSet patch; the unsuspend adds one more only while annotations remain to clear and makes no registry call, where the probe made one per missing pod.
A main is pinned by a required hostname affinity, and neither the drift check nor the migration looked at it once the pod existed: podSpecMatchesAgent ignores affinity, and the migration fast path skips an unscheduled main. A main pending on a full node therefore stayed pinned to it after spec.nodeName moved to another node, which is the usual reason to re-pin. A first boot stayed Pending, and a migration restore that could not schedule kept emitting MigrateNoCapacity for the old target. The migration step now deletes a main that is still unscheduled while its required hostname affinity names a node spec.nodeName no longer names, including a removed pin, and the next pass recreates it under the current pin: the migration restores it from the snapshot when one exists, otherwise the main is created fresh. An unscheduled pod has no VM, so nothing is lost. The check sits before the migration's early returns, so it also covers a set whose phase is not Migrating, and it keys on the hostname term only, which only the operator sets; the recreated pod carries the current pin, so the delete cannot repeat without another spec edit. The check reads the pod only, so a steady-state pass makes no API or registry call.
…ranch, and order hibernation test helpers The collision case named its toolbox "0", so the integer-name check rejected it before the collision check ran; the table case now uses a non-integer name taken by a non-toolbox pod and every case asserts its own error. The three fakeRegistry HasManifest tests become one table, and the hibernation test helpers move below the fixture types.
…rded The suspend marker was a bool, and the unsuspend owed a restore to every recorded VM whose pod was missing, including one an earlier cycle left recorded: a slot deleted by a scale-down before it woke keeps its VM in the record so a later return drops the tag and forks fresh. After one more suspend and unsuspend that VM was owed a restore, so the returning slot restored the earlier cycle's snapshot instead of forking from the current main. The marker is now the list of VMs this suspend recorded, and only a listed VM whose pod is missing is owed a restore; forgetReclaim drops a VM from the list too. No new API or registry call.
…computed from a stale view is corrected patchStatus skips a status equal to the cached CocoonSet's, and the For watch dropped every update that did not change the generation, the operator's own status patches included. Two reconciles of one set a few milliseconds apart then lost the second status: the first patched a status from a pod view that had just gone stale, and the second compared its newer status with a cache that did not hold the first patch yet, found them equal and wrote nothing. Nothing reconciled the set again until another pod event. On hardware an unsuspend that also moved spec.nodeName left the set at Scaling with readyAgents 1 for ten minutes with both pods Ready, in 3 of 13 runs, after vk-cocoon's status writes flipped the sub-agent's Ready condition true, false and true within 15 ms. The For watch now also admits a status change. The patch's own event brings one more reconcile, which reads a cache that holds the patch and rewrites a stale status; an unchanged status is not patched again, so it settles in one pass. Annotation-only patches stay filtered. The cost is one reconcile per status change.
…t status changes reconcile
…attempt, a sub-agent leaves the reclaim record after its create
…redicates setRelevantChange was a hand-rolled predicate.Funcs plus GenerationChangedPredicate; the composed form admits the same events (unset Funcs callbacks return true). Behavior is unchanged; prod -6.
Its own pruning duplicated forgetReclaim's. Both callers run after a successful applyUnsuspend, so Suspended is already empty there, and Restore stays a subset of VMs, so the written record and the write condition are the same. Prod -6.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Residue of the 2026-09-25 family review round on top of main
3004912: every path by which a suspend, unsuspend, image edit, re-pin or migration could leave a:hibernatetag that makes vk-cocoon refuse a pod, strand a suspended state, or park the set inFailed. Each fix is its own commit with a regression test that fails on the code before it. Later commits replace two earlier ones (the missing-slot registry probe, and the guess of a sub-agent's snapshot image from the main pod), so the squashed diff carries only the final mechanism described here.hibernate-reclaimrecord, and lists them assuspended, before the first hibernate annotation. The first unsuspend pass that finds that list owes a restore to each listed VM whose pod is missing, whether or not a pod still carries hibernate, then clears the annotations and the list. This covers passes that clear no annotation, which the old bookkeeping missed:spec.nodeName(the migration moves the main first);Suspendingor a timed-out suspend.source_conflict, dead-letter), and a missing main or toolbox restored without being recorded, so its tag lingered and a later recreate rolled back to the suspend-time snapshot. A VM an earlier cycle left recorded is not listed, so its slot forks fresh when it returns.:hibernatetag pushed from another image drops the tag and boots fresh. Every agent and toolbox create (the main, the released wake, the migration recreate, sub-agents, toolboxes) reads the tag's recorded base image first; on a mismatch it deletes the tag, drops the VM from the record and creates the pod fresh. Before, vk-cocoon refused the create (image_conflict) and the set parkedFailed, or the slot burned its rebuild budget. It costs one manifest GET per managed create.Status.Agentsno longer lists.:hibernatetag alone until the off-target main runs a live VM, so aspec.nodeNameedit during a wake no longer deletes the tag the wake is reading.Migrating.spec.nodeNameno longer names is deleted and recreated under the current pin (restored from its snapshot when one exists). Before, a main pending on a full node stayed pinned to it.HibernateUnsupported) returns to Ready on unsuspend; while a migration is pending it keeps the quiesce instead of looping the migration.patchStatusskips a status equal to the cached one, and theForwatch dropped every update that kept the generation, the operator's own status patches included. Two reconciles a few milliseconds apart then lost the second status: it was compared with a cache that did not hold the first patch yet. The set stayed atScalingwith readyAgents 1 until another pod event, ten minutes on hardware, in 3 of 13 runs of an unsuspend that also movedspec.nodeName. TheForwatch now also admits a status change; annotation-only patches stay filtered.handleFailedMainAgentgodoc states only what the code cannot; theensureToolboxesspec-rejection tests become a table in which each case asserts its own error (the collision case named its toolbox"0", so the integer-name check rejected it before the collision check ran); the hibernation test helpers move below the fixture types.4cb39f9builds the CocoonSet watch frompredicate.Or(GenerationChangedPredicate, predicate.Funcs)instead of a hand-rolled predicate type (unsetFuncscallbacks return true, as the old type did);b5a56e7hasreclaimSnapshotsprune the reclaim record throughforgetReclaiminstead of its own copy of that pruning.Evidence
Two-host kit (host A: control plane and operator, vk node-a; host B: vk node-b), each case an A/B pair on one live stack: the main arm on main
3004912, the residue arm on this branch, arms proven by the sha of the running binaries. The residue cases ran again on the final headd3c8f11, all passing.Failed(image_conflict: the tag was pushed from the old image):hibernateleftScaling, the slot lefthibernatedover its tagFailedsource_conflict)spec.nodeName, a sub-agent deleted while suspendedRunningPending, never recreatedRunningon the new pin:hibernatetag of the sub-agent deleted while suspended outlives the setScaling/1 with both pods ReadyScaling/1 after 30 sScaling/1, thenRunning/2 at the next resourceVersion, 4 ms laterfailed, same guest processThe cut commits on hardware. At
b5a56e7, on a new two-host kit with every open PR of the round combined (this branch, vk-cocoon #113 and #114, cocoon-webhook #25, cocoon #268, all on cocoon-common #35), the 15 Linux cases of the final-head record ran again and all pass: OP-01b, OP-03, OP-04, OP-04b, OP-05b, OP-06, OP-07r, OP-07x (three runs), OP-08, OP-10, OP-11r, OP-12 and OP-13 (the watch showsScaling/1 at rv 1891 andRunning/2 at rv 1892, 3 ms later). OP-02b (macOS) did not run again: its path goes through neither changed function, and the predicate change is exercised by every other case.OP-04 and OP-04b pass on both arms; on main the logs show the migration deleting the tag the wake was reading, but the wake finished first in these runs.
An unsuspend that also moves
spec.nodeNamestalled atScalingfor ten minutes in 3 of 13 runs offee3018(the status staleness above);d3c8f11passed 14 of 14 (OP-07d x10, OP-07x x3, OP-07r). Record chains on the record build (this branch plus cocoon-common #35) pass every case of record in the lanes that hibernate or kill VMs. The chain had written superseded 09-24 bodies over the evidence of seven cases, so they ran again on 09-26 and pass.Gates
At
b5a56e7:make fmt-check;make lintfor GOOS darwin and linux (0 issues.twice);go test -race -count=1 ./...: pass.asl ./...for both GOOS: clean.go build,go vetandgo test -race ./...in a linux/arm64 container: pass.Cost
A pass on a set that never suspended makes no new API or registry call. The first suspend pass adds one CocoonSet patch; the unsuspend adds one more only while annotations remain to clear, and makes no registry call. Each managed pod create adds one manifest GET. Each status change brings one more reconcile, which writes nothing when the status is unchanged.
Size
Prod +247/-58 (the two cut commits: -12), tests +1196/-109, docs +16/-14. Comments +4/-4, each rewritten in place.
Accepted residuals
SuspendTimedOutevery 3 min until the unsuspend or a re-pin; this is unchanged from main.