decisions: production runs on retained releases only; the release is the commit

system/decisions.md

@@ -1068,3 +1068,6 @@ The SFFT rows are runs 145 and 146. Run 146's false positives, 102.7, are one pe
 
 (decision-home-az)=
 **The VPC has one home AZ, us-east-1b, and compute AZs for Batch**: the NAT gateway and the interface-endpoint ENIs sit in the fleet's AZ only (`HOME_AZ` in rapid_systems `cloudformation/gen-lib.sh`), every private route table points at that NAT, and the other AZs (1a, 1c, 1d and 1f: every one that offers the Batch instance types; 1e offers none and MSK may not use it) carry a private subnet, an EFS and a scratch mount target, and nothing else. Both Batch compute environments list all five private subnets; MSK keeps its three brokers in 1a, 1b and 1c, fixed at its creation. Supersedes the per-AZ NAT gateways and per-AZ endpoint ENIs of 2026-07 ("matching the sibling SMDC account"): rapid-db and every host are single-AZ in 1b (rapid_systems `docs/requirements.md` L-single-az-fleet), so redundancy elsewhere could not outlive the database and cost $183 a month. What it buys: Spot and On-Demand capacity pools in five AZs, against the observed failure, `InsufficientInstanceCapacity` on 2026-08-25, 2026-10-31 and 2026-11-02, the last refusing every pair tried in 1a, 1b and 1c; the first smoke job on the Spot lane after the change ran in us-east-1d. What it gives up: internet egress for the 1a and 1c hosts and for every Batch job depends on 1b's NAT (SSM, S3, EFS and MSK do not go through it), and a job outside 1b pays $0.01/GB each way to rapid-db and the brokers, as 1a and 1c jobs already did. Deployed 2026-11-03 01:03 to 01:15 UTC by change sets with no replacement (rapid_systems PR 463); the plan and its Codex pass are in the state vault, `plans/rapid-vpc-home-az.md`. (Ben, 2026-11-03 00:54 UTC)
+
+(decision-production-retained-releases)=
+**Production runs on retained releases only; the release is the commit**: `rapid-rebuild-production`, the one job definition whose role writes products, runs only a rapid commit the retained-releases marker lists. Its identity is `rebuild-<commit>`; its image is one build of that commit, tagged `rebuild-<commit>-<base8>` after the first eight hex of the base image's digest, so the same commit can be built again on each patched base under ECR's immutable tags. Four pieces carry it, all in rapid_systems. `verify-release-retention.sh --retain <release>` keeps a release: each image goes to the backups bucket's no-expiry `archive/` prefix as an OCI archive with its digest preserved, never to a registry, and the marker records each object and digest. The Wednesday `production-rebuild` operation builds production's commit on the newest base and pushes it, registering and repinning nothing. `release-hooks/promote <commit>` alone repins production: it refuses unless the marker lists the commit, promotes the newest build of it, and executes a change set of production's row only; `commit` and `deploy` repin scratch's `rapid-rebuild` and never production. The Friday retention deletes every unpinned build image older than seven days, so a build `promote` supersedes is gone on the first Friday after. Why: Inspector scans every ECR repository in the account and the Science Cloud weekly vulnerability notice counts every digest there, so a release kept in a registry carries its findings onto the notice for as long as it is kept, while a production run must stay on one code release for months inside the patch windows. A promote is due when the operations ledger shows a fixable finding on the production digest against its SLA; a finding in the application's dependencies is fixed by a lock-bump commit on `rapid`, a new release through retain and promote, because the lock files are part of the commit, and the rebuild patches only the OS layer. A promote takes effect at the next processing date. (Ben, 2026-11-04 00:32 UTC; release-retention-cycle tasks 1 to 5, rapid_systems PRs 470 to 474)