[CMGS]

Problem

A CocoonSet with the default retain hibernatePolicy and no spec.nodeName, suspended and then unsuspended, woke its VMs but never deleted their <vm>:hibernate tags. applyUnsuspend only cleared each pod's hibernate annotation.

vk-cocoon's CreatePod restores from any :hibernate it finds (deriveRestoreFromEvidence, vk-cocoon #55). So a later recreate of the pod silently rolled the guest back to the suspend-time state, leaving only a HibernateSnapshotExists warning. Recreates happen on a force delete, a failed-main rebuild or a drift rebuild.

The other wake paths already drop the tag once the VM is live: the CocoonHibernation wake (093eac8 names this exact rollback), the release-policy wake and a migration. The 2026-09-24 two-host round saw the tag survive a plain suspend and unsuspend.

Change

• Before applyUnsuspend clears any pod annotation, it records two things in cocoonset.cocoonstack.io/hibernate-reclaim: the set's generation, and the names of the managed VMs it wakes.
• Every pass that reaches the unsuspend step deletes a recorded VM's tag once its pod is woken, then drops that VM from the record. Woken means all of these hold:
  • the pod no longer carries the hibernate annotation;
  • it runs a live VM;
  • vk-cocoon reports lifecycle-state=ready at the recorded generation or later.
• A pod that is missing, terminating, terminal or not yet woken keeps its VM recorded. That covers a pod rebuilt mid-wake, which restores from the tag and is reclaimed once it wakes.
• A delete the registry refuses keeps the VM recorded and fails the pass, so the controller retries with backoff, as the CocoonHibernation wake does. Progress on the other VMs is saved first. Without the retry, a force delete arriving before the next event would recreate the pod over the tag that was still there.
• The annotation is written only when its value changes, and it goes away with the last VM.

The ready-at-the-recorded-generation gate means a stale pre-suspend ready in the informer never passes. So does an unsuspend that lands mid-hibernate, while the VM is still live and vk reports hibernating. A spec edit after the wake does not hold the reclaim back.

Cost

• A set with no record costs one map lookup per pass: no registry call and no JSON decode.
• Each woken VM costs one registry probe and one delete, once.
• A VM whose slot left the spec stays recorded until the set is deleted. That costs no registry calls.

Tests

cocoonset/reconciler_test.go:

• applyUnsuspend merges the woken VMs into an existing record at the new generation. It writes the record before any pod patch. A second pass over a stale pod cache does not rewrite an unchanged record.
• reclaimWokenSnapshots table, one row per case:
  • woken at the unsuspend generation;
  • still hibernated;
  • ready before the unsuspend;
  • ready without a live VM;
  • hibernating again in vk;
  • rebuilt mid-wake;
  • a spec edit after the wake;
  • one VM woken and one not;
  • the registry refuses.
• A full Reconcile deletes the woken VM's tag, clears the record, and makes no registry call on the next pass.

Each of these mutations fails a test: dropping the live-VM, ready-state or annotation check; gating on the set's current generation; stopping at the first unwoken VM; recording after the clears; rewriting an unchanged record; dropping the prior record.

Gates, all with GOWORK=off:

• make fmt-check
• make lint on darwin and linux: 0 issues. for each
• go test -race -count=1 ./...
• asl ./... on darwin and linux: clean

Docs

docs/cocoonset.md steps 6 and 10 describe the record and when a tag is reclaimed.

Commits

• 21c9b2d the fix.
• ede2d84 a refused reclaim fails the pass so it retries, and docs step 10 corrects the removed-slot case.
• cdeedd2 review only: own-line comments in suspend.go and reconciler.go start uppercase with one fact each, and the reconciler tests reuse mustGetCS.
• abcc20c review: the reclaim record uses encoding/json/v2; its bytes equal v1's, so a record an earlier build wrote reads back unchanged.
• 9bf876c review: the rebuild history moves to encoding/json/v2 too (Deterministic(true) keeps sorted keys, omitzero keeps an unparked entry's parked out), so the repo imports one JSON package. A new test pins the exact bytes.
• 9c072a0 review: own-line comments start uppercase unless they open with an identifier, repo-wide.
• 6b3944d review: docs step 10 states what a refused reclaim holds back: the rest of that pass, scaling and the status patch included.
• 99606a6 test: the VM-lock tests passed with the lock disabled, because no fixture put two registry calls in flight. They now check with TryLock that the VM's lock is held at each registry call, on fixtures that reach the registry. The deleting-versus-live case never reaches the registry once its CRs carry UIDs, so it is dropped; the retargeted case covers the delete-path lock. Disabling either lock fails a test.

Hardware verification

Verified on 2026-09-25 at 21c9b2d on two bare-metal hosts, as an A/B pair against the same live stack. H-06 and H-07 fail on main and pass on this branch. ede2d84 changes only the refused-delete path, which the unit tests cover; the hardware round did not inject a registry failure. Results: #60 (comment)

[CMGS]

Hardware verification on 2026-09-25, on two bare-metal hosts with the two-host E2E kit. Every kit binary, the firmware and the running operator, webhook and vk-cocoon processes were checked by sha256 before and after each lane.

Both arms ran H-06 and H-07 against the same live stack. Only the operator changed between them: main 4d4e61c (sha256 c5220cdb), then this PR at 21c9b2d (sha256 97124416), then main again.

Each case creates a CocoonSet without nodeName and suspends it. The phase reaches Suspended only after vk reports this round's hibernate, and :hibernate is then in the registry. The case unsuspends the set and waits until the guest answers again.

case	main 4d4e61c	this PR 21c9b2d
H-06: :hibernate is gone within 120 s of the wake, and the reclaim record is cleared	FAIL. The tag was still in the registry 120 s after the wake.	PASS. The tag was already gone at the first check after the wake, and the record was cleared.
H-07: marker 1 is written before the suspend and marker 2 after the wake, then the pod is force-deleted; the recreated pod must boot fresh	FAIL. The recreated pod restored the suspend-time snapshot. Marker 1 was back, marker 2 was gone, and the new pod had one HibernateSnapshotExists event. Everything the guest wrote after the wake was lost.	PASS. The recreated pod booted fresh, with neither marker and no HibernateSnapshotExists event.

The wake itself brought marker 1 back on both arms. After the PR arm, main was restarted and the version gate passed again.

Report and evidence: cocoonstack/cocoon-specs tests/2026-09-25-two-host-rerun.md at fb74e17, with one evidence file per case under tests/2026-09-25-two-host-rerun-evidence/.

[CMGS]

Record-build chains, 2026-09-25 afternoon. 99606a6 ran on both hosts:

• H-06, H-07: PASS.
• H-08: PASS. The registry was restarted read-only (storage.maintenance.readonly) across the unsuspend. All 15 of the operator's DELETEs got 405, the VM stayed in the reclaim record, and the tag was reclaimed once deletes came back (202).
• The chaos lane passed 11/11.

The two earlier H-08 runs were kit defects and are voided in the round report:

• v1 ran the registry with storage.delete.enabled: false, which distribution 3.0 still answers with 202 for a tag delete, so no refusal ever happened.
• v2 put maintenance.readonly at the top level, where it is ignored; the probe got 404 and the case stopped.

(Edited: an earlier version of this comment described the two voided runs incorrectly.)