fix(server/azure): exclude uninitialized replicas from readiness

[cristim]

Summary

Azure readiness admits HTTP server processes before their lazy database initialization completes. A successful warm-up request does not establish readiness across replicas. This caused the main-branch deployment smoke test to fail after PR #472 merged at 4fe935e3dee6c0a7464cda279fbc308cbca728d5.

This is separate from #472's AWS recommendation completeness change. The startup, health, workflow, and Azure compute files traced here are unchanged between reviewed head ed80d762cd269b45482f57296306611d966beb8b and that merge.

Current behavior and evidence

Azure run 37243528487, Test Deployment records:

• 2026-10-04T23:29:33.116831115Z: the health gate returns healthy, including auth, config, and migrations, after an API warm-up.
• 2026-10-04T23:29:33.283074595Z: the first smoke request returns degraded, with auth store uninitialized, config connection pending, and migrations not yet run. The interval is approximately 167 ms.
• Deployment succeeded; smoke verification failed. The diagnostic log step was skipped.

Confirmed source behavior:

• .github/workflows/deploy-azure.yml:446 warms /api/auth/check-admin; :469 smoke requests only /health and fails on the first degraded response.
• internal/server/http.go:31 routes /health without ensureDB; API requests initialize through :162.
• internal/server/app.go:485-493,530 creates the uninitialized state. :605-666 initializes it; successful initialization does not reset it to pristine state.
• internal/server/health.go:58-64 returns HTTP 200 while degraded.
• terraform/modules/compute/azure/container-apps/main.tf:165-173 uses /health for readiness, admitting those processes.

Inference: the second response came from another cold replica or a restarted process. The logs lack replica identity, so they do not distinguish these cases. This is not evidence of a failed migration: that state reports failed, not pending.

Reproduction and expected behavior

Locally, route requests across two independent cold application instances. Warm one through the API, then send /health to the other. The current design admits the second instance and returns degraded HTTP 200. This proposed local reproduction has not yet been executed.

Every instance eligible for traffic must have completed required initialization. Deployment verification must continue rejecting degraded responses.

Proposed fix and verification

Initialize each HTTP-serving process independently of user traffic in internal/server/http.go and the existing initialization path. Add a dedicated readiness handler in internal/server/health.go, returning non-2xx until required checks pass. Preserve liveness separately. Wire Azure readiness and workflow verification to that handler.

Test the two-instance scenario, permanent initialization failures, failed migrations, and bounded smoke-test failure. Returning non-2xx alone needs an initialization trigger; adding sleeps or warm-up requests cannot guarantee every replica initializes.

Related work and severity

#438 covers script tests and failure logs. #31 covers deploy-time migrations. #217 established the healthy-body assertion; preserve it. A fresh open-issue search for readiness OR "cold replica" returned no matches; these related issues do not explicitly cover this readiness contract.

Severity: high. The default-branch Azure deployment is red, and readiness does not represent dependency readiness. Proposed labels, subject to existing-label verification: priority/p0, type/bug, severity/high, urgency/now, impact/many, effort/m, triaged.

[cristim]

Taking this P0 through a scoped fix. Source investigation confirms that readiness currently accepts degraded HTTP 200 and initialization depends on traffic to each process. The plan will preserve liveness, initialize each HTTP process independently, and verify two independent local instances plus failure cases on macOS. Synchronous startup would change probe timing across other deployment targets, so the plan is checking a narrower asynchronous initialization approach for race safety before implementation. Independent Astra review and exact-head local verification plus green CI remain required. No cloud deployment or credentials are part of local verification.

[cristim]

Another local session has a verified implementation for #488 in /Users/cristi/.claude/worktrees/platform488-readiness-20261005, based on ed44785a. Nine files, 660 changed lines including 486 test lines, no commits or PR yet. Native PostgreSQL/HTTP tests, parent-red and two mutations, race/build, actual 600-second workflow timing, pinned Terraform and two independent staged reviews are retained under evidence/platform488/. A final complexity correction is verified; the current index retains the preceding reviewed tree plus an unstaged cleanup-helper extraction.

We discovered the shared plan is now owned by /root/platform488_readiness_investigation in cloud-commitments-platform-488-readiness, consistent with claim comment 5991176944. Our source is isolated and preserved; we have stopped publication and shared-plan writes. Please coordinate through this issue before either session opens a duplicate PR. The claimed owner should lead publication; compare both implementations and reuse the stronger verified behavior and evidence rather than discarding either tree.

Our implementation differs in startup policy: it retries failed initialization with a five-second delay, while the current owner's plan explicitly fails the server after initialization failure. It also uses an RWMutex dependency snapshot, while the owner's design installs a startup completion barrier before public requests can win initialization. These contracts need deliberate reconciliation against the issue before selecting a base. No claim is made that the two designs are equivalent.

The current user's strict 400-line PR rule also requires a publication split. Our proposed stack separates the approximately 300-line implementation and Azure wiring from the 360-line native lifecycle regression file, with both stacks verified before merging and issue closure after both merge. No split has been committed or published.

[cristim]

Independent cold comparison recommends the claimed owner's implementation in cloud-commitments-platform-488-readiness as the base. Our alternative remains preserved and unpublished.

Two concrete defects disqualify our alternative: API/scheduled requests can win initialization before the startup worker and give the process-wide cleanup worker a request context; deferred signal unregistration can swallow a second SIGTERM while migration joining blocks. The owner's completion barrier and early signal unregistration address both. Do not import our infinite startup retry policy.

Required correction in the owner's tree: deploy-azure.yml's five attempts allow approximately 50 seconds for immediate readiness failures, or 100 seconds if every curl times out. Existing database connection retries permit 105 seconds before migrations. Adopt bounded deadline polling with a justified budget and execute the actual workflow blocks against delayed success and permanent failure. Our retained 105-second delayed-success and 600-second failure evidence can inform the fixture, but must be reproduced on the selected final code.

Remaining verification: rerun the owner's native shutdown witness with child-only GORACE=atexit_sleep_ms=0 and reproduce a missing-initializer-join mutation failure. Preserve the explicit context-free migration cancellation limitation and follow-up #498. No cloud proof is implied.

Strict publication limit: cold comparison counts 877 changed lines in the owner's tree including untracked files. Split into independently reviewable PRs of at most 400 changed lines, with complete regression evidence and exact-head independent review for the final stack before merge.

Revision evidence: alternative base ed44785, changed-file digest 62692250f1dab58d4cbeae8d05e32f3dfefdd8b5b67e2c6118851c8d5b60d9da; owner base a442002, changed-file digest 21f852a4bcf17dbf581cf838a4266a57c66cdc24153157b4814f0e49d89a8a98. This comparison was read-only source review, not committed-head runtime acceptance.

Please acknowledge ownership coordination here and carry these concrete corrections through the owner's implementation. This session will not edit or run that tree without coordination.