[Ruinan Liu (ruinan-liu)]

Summary

Add a background garbage collector to the tombstone recovery operator. While the reconciler is running, it periodically deletes stale inventory Secrets from the downstream cluster's kube-system namespace.

Behavior

• Registered with the controller manager when tombstone recovery is enabled, runs only on the leader replica, and starts a cycle every 3 to 4.5 minutes.
• A failed or panicking cycle is logged; the next cycle starts fresh with no in-memory progress.
• Each cycle lists inventory Secrets with the eno.azure.io/inventory-lineage label in one call, then lists Compositions and Symphonies in the reconciler's namespace with the uncached API reader. Any read failure skips the cycle.
• Runs only when the namespace's Symphony exists and is not deleting:
  • Composition is gone: delete every Secret whose lineage matches no Composition.
  • Composition exists: delete Secrets from older synthesis UUIDs, only once the current synthesis UUID has a complete snapshot.
• Deleting Compositions and Compositions without a current synthesis keep all their Secrets.
• Logs each deleted Secret and a per-cycle summary of scanned, deleted, and failed Secrets. A failed delete is retried in the next cycle.

Safety

• Symphony deletion orphans downstream resources and keeps their inventory on purpose, so nothing is deleted while the Symphony is deleting or missing.
• Compositions are listed before the Symphony, so a Composition missing from the list was gone before any observed Symphony deletion.
• Older snapshots stay until the current snapshot is complete, so recovery keeps its history during recreation and resynthesis.
• Deletes use UID preconditions, and NotFound counts as deleted.
• No finalizers, no Composition status writes, and no downstream resource deletion.

Refactor

• Add deleteInventorySecret, shared by inventory recording rotation, deletion cleanup, and garbage collection. Deletion cleanup now also uses a UID precondition.

Assumptions and Requirements

• Only this reconciler writes inventory Secrets to its downstream kube-system.
• Each namespace has one Symphony, and each Composition has a Symphony owner.
• The reconciler needs upstream list permission on symphonies.

[xiazhan]

A comment not specific to this pr: Do each Eno controllers/reconcilers well support graceful termination?

[Ruinan Liu (ruinan-liu)]

REspond to comments

[Ruinan Liu (ruinan-liu)]

A comment not specific to this pr: Do each Eno controllers/reconcilers well support graceful termination?

Both Eno binaries use controller-runtime's single handler, so SIGTERM cancels the manager ctx and stops all controllers and runnables including the new inventory gc. Leader election releases the lease on cancel. I will confirm that every controller drains the in-flight work within the pods' termination grace period.