Skip to content

Qualify pinned producer and direct MDM idle delivery for #11 - #14

Draft
grove wants to merge 66 commits into
mainfrom
issue/11-explicit-escalation-time
Draft

Qualify pinned producer and direct MDM idle delivery for #11#14
grove wants to merge 66 commits into
mainfrom
issue/11-explicit-escalation-time

Conversation

@grove

@grove grove commented Oct 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Owner approval: S1 59/min acceptance floor — 2026-10-22

The requester explicitly adopted 59 cases/minute as the minimum passing S1 throughput, while the frozen 300-admission workload and 60/minute offered rate and engineering target remain unchanged. Only the sole frozen throughput failure in [59,60) with original raw exit 1 and all other original gates passing can receive a disclosed owner-approved-floor result; less than 59, other failures and incomplete population/receipt/resource/latency checks still fail.

pg-trickle #1168 and React #12 record the decision; dependent pg-trickle PR #1213 implements the checker and React plan PR #22 adds the dated spec addendum. Historical 59.363743/min output at pg-trickle 2d102645 meets only the new numerical floor for that tested image; it is not final source-image qualification. Latest real candidate/control benchmark on approved 10-CPU ARM64 and React clone/restore, signed operating envelope and release tests remain outstanding. Keep PR #14 draft; no merge to main or activation.

v0.49 owner review and further qualification — 2026-10-22

  • Owner review: the requester reports that they have reviewed and approved PR Qualify pinned producer and direct MDM idle delivery for #11 #14. This is an explicit user statement, not a submitted GitHub review or a signature on the required quantitative operating envelope. The PR remains draft while the separate release gates are incomplete.
  • New merged dependent: PR #20 was integrated into issue/11-explicit-escalation-time as 6a55008 after its source-head checks passed. It proves public pgreact.claim(max_items=1) leaves another independent pending episode available for a separate claim, and both claims execute once. The merged head's fresh pinned-producer CI remains pending and must independently pass before the current combined branch is certified.
  • Next dependent draft: PR #21 at 24fbee3 checks the installed pgreact_internal.managed_cycle → bounded pgreact_api.claim function chain and records exact function hashes and managed heartbeat/processed deltas during the two-case outage. It also corrects an inaccurate GUC help string. Pinned CI is pending. This is not yet accepted or merged.
  • Open release blockers: The S1 300-case >=60/min source-qualified paired throughput gate pg-trickle #1168 remains open; PR #1191 has a failing refresh test and an unfinished safety/benchmark profile. Coherent restore and clone preactivation isolation, mixed-workload backlog fairness, full T49.01–T49.12/C01–C12 evidence, numeric limits and owner-signed operating envelope also remain. This PR is neither safe to merge into main as an accepted v0.49 release nor authorized for production activation.

Backlog evidence integrated and exact-head green — 2026-10-22

  • PR #19 merged into the feature branch as d70a73d. Its two-case outage test records a monotone completed-case/receipt progression timeline, wall-clock drain duration, worker pending/processed status and unchanged full durable state after later worker cycles and a cold restart.
  • All merged-head checks green: pinned joint MDM producer #39681815208, MDM preview #39681815435, core/adapter #39681815413, PR CI #39681815490 and push CI #39681807789.
  • Acceptance limits remain: Recording public pending/processed counts with batch_size=1 is not proof of at-most-one admission per cycle; mixed-workload backlog fairness, crash/uncertain-commit, clone/restore activation barriers, signed 300-case S1 resource/latency envelope and exact current-source pg-trickle #1168 60 cases/min qualification are still outstanding. No release, no merge to main, no automatic production escalation.

Two-case real post-outage drain merged — 2026-10-21

Post-merge three-case routing qualified — 2026-10-21

Three-case sampled-time routing qualified and integrated — 2026-10-21

Merged one-case outage regression — 2026-10-21

Post-resume cold restart — new head 4cda34f (pending CI)

React feature commit 4cda34f9f9c578c55851498d62767f9b3fb9f9a8 adds a new end-of-audit recovery check after the validated pause→restart→authorized resume→stale-version rejected operator sequence. It restarts PostgreSQL again, requires a new postmaster and multiple real managed-worker cycles, and compares complete MDM case/receipt, immutable React intent/request/attempt/work, business/technical source and exact MDM+React binding/runtime vectors. The active binding must retain its exact post-reconciliation runtime version, and the previously rejected old-version calls must not generate delayed effects.

The previous exact head 7a95232 is GREEN: pinned MDM #39628593830, full repaired-MDM preview #39628593800, core #39628593915, adapter #39628593943. These exact-image tests qualify the repaired MDM dependency in the feature branch but are not a release or production activation. The new restart test is pending its own head checks. The test still enrolls only one selected case, not autonomous recurrence; it does not resolve outage catch-up, live claimed-episode races, clone isolation, signed release limits or React #12.

MDM stale-runtime repair adopted in feature branch — 2026-10-21

The earlier handoff below is kept as historical chronology.

Current cross-project handoff — 2026-10-21 (source repair reviewed separately)

The older chronological log below retains its original as-observed historical CI status; use this handoff for the latest verified position.

Exact-source artifact evidence preview — current head 20f09de (pending)

The earlier full repaired-MDM React preview #39582414674 passed on preview source 261bb41; the untouched original MDM-pinned CI #39582414884 failed as expected at the stale-version bug. Neither was represented as a production pin adoption.

The corrected MDM draft PR #21 now has line-neutral source commit 6ad23ca: the guarded lazy SPI row read preserves later SQL annotation locations so MDM's strict frozen v0.14.1 archive byte comparison is not weakened. The prior 261bb41 MDM Rust/test CI passed fmt/clippy/tests and failed only the generated SQL archive cmp at embedded source line 822; unchanged MDM main #39581225844 passed. Current MDM CI #39592063438 is queued, so this archive repair is not yet qualified.

New React head 20f09de updates only the separate MDM-candidate preview to pin 6ad23ca and additionally capture exact candidate MDM Git source-archive SHA, packaged binary SHA-256 hashes for pg_mdm.so, pg_trickle.so and pg_react.so, raw image inspection, and a manifest digest. The step compares the actual MDM and pg-trickle shared libraries byte-for-byte across the candidate MDM image and dependent React image. Exact preview CI #39592436407 and core checks are queued. The original pinned workflow and historical producer checksum are unchanged. Do not merge, repin or enable production before final-head MDM and candidate joint-stack CI succeed, with explicit artifact adoption.

Exact MDM repair preview head — 2026-10-21

Preview-only React head 6266248 pins the separate MDM repair branch's latest exact source 261bb419447bb2bbebd3bce5be514baa4ed3a1e4 (draft MDM #21) after a source-formatting cleanup. Both current exact-source MDM CI #39582365065 and React preview qualification #39582414674 are queued. The original pinned-producer CI #39582414884 remains unchanged and expected to reproduce the original error. The producer pin is not adopted, and no passing patched-artifact result is claimed.

New isolated preview for corrected MDM source — 2026-10-21

React PR head 7db1bf4 adds .github/workflows/mdm-stale-cas-preview.yml, a separate end-to-end job for the exact MDM error-path repair in draft MDM PR #21, pinned to MDM source 261bb419447bb2bbebd3bce5be514baa4ed3a1e4. It builds the modified library with the original historical pinned pg-trickle producer d8bb8eb5d45c5c9982e173e88914037433a1ee05 and archive checksum b8d4c59ca3770c305659255e09adde303b4d220228392c415ae363aa7be0a2da, then builds React, runs the full audit and requires actual stale-version negative receipts/errors plus the earlier idle and binding tests. It preserves exact MDM/React source file digests, image inspect metadata and runtime evidence as a CI artifact. Preview CI #39582038003 queued; no pass is claimed.

This preview is NOT a replacement pin or a release artifact. The original untouched MDM-pinned CI #39582037963 remains a strict test of the previous source and is expected to expose the original SPI internal-error regression until a separately qualified adoption. Keep the failure, correct-code draft, and full evidence side by side. MDM issue #20, draft MDM PR #21 and MDM CI #39581517997 track the blocking repair. Do not merge React or enable production on preview evidence alone.

Current blocking upstream defect — 2026-10-21

React head 34c06f76075bfb50bb0ba572b02a32baa1af0157 is NOT green. Pinned-producer integration #39572339881 failed in resume-stale-version.sql:109; core #39572340077, legacy adapter #39572340055, and push CI passed. The failure occurs only after earlier real idle delivery, cross-key human edits, MDM row-lock concurrency, genuine recurrence, maximum-level safety, pause/restart, authorized resume and worker-quiescence checks all pass. A stale operator call to pgreact_mdm.pause_intent_binding returns XX000 MDM_INTERNAL: PostgreSQL SPI failed: SpiTupleTable positioned before the start or after the end, not the expected MDM_POLICY_BINDING: runtime version is stale or binding is replaced.

Cause identified: in the pinned MDM 0.14.1 artifact, src/api/policy_intent.rs tries row.first().get(1) on the zero-row INSERT … ON CONFLICT … WHERE runtime_version = expected RETURNING result, preventing its intended fail-closed error. Tracked in MDM issue #20; targeted draft MDM PR #21 adds row.is_empty() handling and MDM E2E negative assertions for both stale pause/active calls, including unchanged complete runtime/operations/case/receipt state. MDM CI #39581517997 is queued.

Do not relax or mark React's new stale-version test passing. Preserve the exact pinned MDM source 06471e647ee54a80973fb5ad6709830f9955dbff, producer archive checksum and old red evidence. A corrected packaged MDM binary and React joint-stack qualification must be separately built, identified and checked before adopting it. This PR remains draft; #11/#12, signed throughput envelope, missed-deadline catch-up and restore/clone gates remain open. No producer repin, privilege broadening, or production activation is authorized by this work.

Stale operator runtime-version fencing — head 34c06f7 (CI pending)

Latest commit 34c06f76075bfb50bb0ba572b02a32baa1af0157 adds and invokes time/resume-stale-version.sql, after the already-green MDM binding pause→cold-restart→authorized resume. An actual non-superuser MDM entity execution role tries both pause_intent_binding and reconcile_intent_binding using the old paused runtime version while the binding is already active at the newer version. The fixture requires both calls to fail on MDM's stale/replaced binding-version fence, records each SQLSTATE and message, and compares the complete MDM and React binding/runtime populations, MDM cases/receipts, immutable React intent/work/attempts and business/technical sources with their pre-attempt state. The active runtime version must remain unchanged. The fixture's auxiliary test function is SECURITY INVOKER and exists only inside the disposable integration database.

The previous exact head a043de1 passed pinned-producer CI #39565839468, core #39565839478, adapter #39565839500, and push CI. The stale-version addition is not yet qualified: pinned-producer CI #39572339881 and core CI #39572340077 are queued. No production enable, privilege broadening, original producer pin swap, or release sign-off; PR remains draft.

Authorized binding resume after cold restart — new head a043de1 (CI pending)

Commit a043de15e84e88c2d84f41a7e1860395d7b31653 adds time/resume-after-pause.sql and invokes it after the already-green operator pause and cold-restart barrier. The authenticated MDM entity execution role calls pgreact_mdm.reconcile_intent_binding using the exact paused runtime version, and the test requires the MDM state and React active-runtime record to become active at the newly advanced version. The original pause-denied key must still replay its exact immutable BINDING_PAUSED receipt, but a new key on that active binding must produce LIMIT_EXCEEDED / ESCALATION_LIMIT at its unchanged maximum level 2; a retry must replay this different terminal receipt. All original receipts, MDM cases, immutable React intent/attempt/work records and business/technical sources remain intact. Multiple subsequent real managed-worker cycles must also leave complete case/receipt/adapter/runtime/source vectors unchanged.

This is operator pause→restart→authorized resume correctness for the existing enrolled single-case rule; it does not qualify pending React episode withdrawal, autonomous second-case scheduling, a downtime period spanning new due deadlines, restore/clone, or full #11/#12 acceptance. The previous exact head 6ab404e passed pinned-producer CI #39561580441, core #39561580447, and legacy adapter #39561580464. The new head is unqualified pending pinned-producer CI #39565839468, core #39565839478, and adapter #39565839500. Keep PR draft and production off.

Paused-binding cold restart — head 6ab404e (pending)

Commit 6ab404ecdb80b91ad73de3be027349797575e23f extends the authorized operator-pause fixture to a real PostgreSQL restart. It snapshots all MDM cases/receipts, React immutable intent requests/attempts and work, business/technical source rows, plus complete MDM and React binding/runtime records. It requires a new postmaster, a ready managed worker that advances through multiple later heartbeat cycles, exactly identical state/bindings before and after, and continued absence of the paused binding from the React active-runtime table. This is durability of the operator pause barrier for the previously enrolled single-case rule; it does not test stale pending React claims, clone activation, restored databases or outage catch-up.

The prior 8aa8bf8 passed pinned producer #39548527426, core #39548527505 and adapter #39548527433. The new restart proof is not yet qualified: pinned producer #39561580441, core #39561580447. Keep PR draft and production disabled.

Operator pause-barrier safety — head 8aa8bf8 (pending)

Commit 8aa8bf8f83bc65a75708140587ebdf28c76dfce0 adds an end-of-audit authorized binding pause fixture after all the earlier green idle, human-edit, concurrency, recurrence, maximum-level, and restart checks. It calls pgreact_mdm.pause_intent_binding as the actual MDM entity execution role with the current runtime version, checks that the adapter's runtime binding is removed and MDM's runtime state becomes paused, then attempts a new authenticated ESCALATE request as pgreact_mdm_worker. The expected durable result is BINDING_PAUSED / BINDING_PAUSED with exact canonical request/digest, unchanged full MDM cases, immutable React/source vectors, and a matching same-key receipt replay. No privileges or existing production artifacts change.

Prior head cbe4882 passed pinned producer #39502053389, core #39502053204, and adapter #39502053274. New head remains unqualified: pinned producer #39548527426 is queued. This is an MDM authority pause-barrier probe; it does not prove React's eventual work withdrawal, backlog recovery or clone isolation. Keep PR draft, #11 open, production disabled.

Post-limit restart recovery — latest head cbe4882 (CI pending)

New React commit cbe488261d4dfb6d0671433e1dbebc4cebd56698 extends the already-green maximum-level quiescence test to restart the PostgreSQL postmaster after the denied level-3 attempt. The audit requires a new postmaster epoch, a ready legitimate worker advancing across multiple cycles, identical full durable MDM cases/receipts, React immutable intent/attempt and work state, and business/technical-source vectors, including exactly one recurrence APPLIED_CONTROL receipt and one terminal LIMIT_EXCEEDED / ESCALATION_LIMIT receipt. No added grants or production policy activation.

The previous head 454c205 passed pinned-producer CI #39495741221, core #39495741351, and legacy adapter #39495741305. The new restart assertion is not yet qualified; pinned-producer CI #39502053389 and core CI #39502053204 are queued. Scope is retained-case recovery of the enrolled single-case worker, not overdue backlog after outage, autonomous recurrence discovery, clone/restore, or final #11/#12 release acceptance.

Post-maximum managed convergence — pending on 454c205

React commit 454c2053799ec6a264e86198af55dddf0391731a extends the passing maximum-level authority test to observe multiple later cycles of the real managed worker, then compare complete MDM cases, all receipts, immutable React request/attempt and work state, and business/technical source vectors. It requires exactly one previously applied recurrence level-2 receipt and one retained terminal level-3 limit-denial receipt. This proves only no delayed effects for the already enrolled single-case rule after the authority limit denial; it does not establish autonomous recurrence scheduling, multi-case admission/backlog bounds or outage catch-up.

Exact new-head pinned-producer CI #39495741221 and core CI #39495741351 are queued. Previous head 85ec4ddb passed pinned CI #39484595864, core #39484596006 and adapter #39484596170. Remain draft; no production activation or release sign-off.

Recurrence maximum-level boundary — latest head 85ec4dd (pending)

Commit 85ec4ddb28e8a5c582a608e8e4d699bbe209ed33 adds an exact MDM authority-level limit test after the real recurrence and postmaster-restart checks. The enrolled binding's maximum is exactly level 2. A fresh, authenticated attempt to advance that new occurrence to level 3 must return a durable LIMIT_EXCEEDED / ESCALATION_LIMIT receipt with an unchanged entire case population, immutable React work/intent state, source vectors and original receipt history. A same-key retry must replay the denial without any second receipt. This checks MDM's enforced binding limit in the React joint stack; it does not demonstrate React admission bounds, missed-deadline catch-up, or multi-case worker scheduling.

Prior head 396bdbab passed pinned-producer CI #39465133572, core CI #39465133760 and legacy adapter #39465133588. The new maximum-level fixture is pending current-head CI. Keep this PR draft, #11 open, and production activation disabled.

Post-recurrence restart regression — current head 396bdbab (pending)

Commit 396bdbab75de51a862a8fb27786a42d6472890e5 extends the already passing genuine MDM recurrence and post-recurrence idle quiescence audit: restart the real PostgreSQL postmaster, witness a new postmaster startup, wait for the legitimate managed worker to resume and complete multiple later cycles, then compare the complete persisted MDM cases/receipts, immutable adapter requests/attempts, React work/attempts and business/technical source rows before versus after. Separately require the old and new real MDM occurrence IDs, one applied level-2 receipt for the new occurrence, and zero new effects from replay. This is post-delivery restart safety for the enrolled single-case rule and existing recurrence, not autonomous discovery of the second case, an outage across undelivered deadlines, clone isolation, or coherent restore.

Exact new-head CI: pinned producer #39465133572, PR core #39465133760, legacy adapter #39465133588. All are pending; do not count the new regression as passed until those results are checked. Previous d8b7fb6 head passed its relevant qualification runs. The draft/production/release gates remain unchanged.

Post-recurrence managed-worker regression — passed on prior head

Commit d8b7fb6 adds an observation-only check after MDM's genuinely reopened occurrence has independently applied level 2. The ordinary React background worker must remain ready for subsequent heartbeat cycles while complete case/receipt/immutable-intent/work/business/technical-source vectors remain identical. The registered React rule is still scoped to its preselected case: this test does not prove autonomous discovery, cross-case catch-up, or clone/restore safety. Pinned-producer CI #39454625856, PR core #39454625947 and legacy adapter #39454625906 passed on d8b7fb6. New restart extension at 396bdbab is separately pending.

Recurrence fixture correction — 2026-10-20

Latest head d413f72: PINNED-PRODUCER RECURRENCE GATE PASSED. CI #39451120393 passed and its log confirms recurrence-independent; push core CI #39451109531 passed. This is the focused MDM-authority recurrence fixture, not autonomous React worker discovery. The prior heads 68918d7 and c837f8f failed pinned-producer CI #39442250486 and #39442533637 because recurrence fixture SQL referenced public.e2e_policy_vars, a temporary E2E support table that upstream MDM deliberately drops before foundation is cloned. Every preceding idle, human-reduction, concurrent-key and stale-token test passed before that missing relation was accessed. No new occurrence control was attempted on those failed runs.

The correction derives resolved occurrence 1 / reopened occurrence 2 directly from MDM's retained public policy_cases_v1 rows (entity_name, issue_key, occurrence, open/resolved state, review ID). It fails closed unless exactly one public eligible pair exists, printing the retained case-state vector on mismatch; it does not recreate private fixture tables or synthesize recurrence identity. Subsequent positive one-level application, unique receipt/replay and unchanged old-case/React/source assertions remain intact. Latest pinned-producer CI #39451120393 passed with the original eligibility, receipt/replay and state-preservation assertions intact. The remaining R3 and v0.49 release gates are separate.

Earlier v0.49 evidence and recurrence test scope

Prior exact head f81ce2d88b917958b098f3db2cab428cecd68a12: GREEN. Pinned joint-stack CI #39428111533, PR core CI #39428111573, legacy adapter #39428111724, and push CI #39428103975 all passed. This specifically qualifies the distinct terminal outcomes after human change (old tokens → STALE_CASE / FRESHNESS_TOKEN_MISMATCH; current tokens for a previously applied level → ACTION_DENIED / ESCALATION_LEVEL_ALREADY_APPLIED) with complete case, receipt, replay and React/source vectors.

Earlier recurrence fixture candidate c837f8f: FAILED BEFORE CORRECTION. The added time/recurrence-independent.sql reuses MDM's actual resolved occurrence 1 and reopened occurrence 2 for the same retained issue_key from upstream e2e_policy.sql; it requires separate case/review identity, a currently open eligible new occurrence and no prior automated level-2 application on that new case. An authorized human gives the new case a past due time and level 1; the authenticated MDM worker must apply level 2 once and return the identical receipt on an original-key retry. Complete original case/receipt history, exact request-body/digest and post-control row, and unchanged React work/intent and business/technical-source vectors are asserted. This is a real MDM recurrence boundary within the React joint stack, not evidence that the React managed worker autonomously discovered that newer occurrence or that cross-occurrence after prior applied2 has been qualified. The same acceptance checks subsequently passed on corrected head d413f72.

Status: Keep PR draft, #11 open and production activation disabled. Signed operating profile, genuine worker-claim races, bounded catch-up/recovery/clone and React #12 / pg-trickle #1168 remain open. The post-merge pg-trickle main CI and TPC-H failures are independent of the passing React pinned-original-producer runs.

T49.09 freshness-after-human-edit regression (qualified on previous head)

  • f81ce2d88b917958b098f3db2cab428cecd68a12 adds the exact time/human-stale-after-concurrency.sql fixture and runs it after the previously passing idle/cross-key/row-lock tests. It captures an authorized MDM case's full pre-edit state, toggles manual queue-protection through mdm_steward.set_case_controls (raising action_revision), and then submits an old captured level-2 ESCALATE request and a fresh level-2 request. Expected: old token → durable STALE_CASE / FRESHNESS_TOKEN_MISMATCH; new token → durable ACTION_DENIED / ESCALATION_LEVEL_ALREADY_APPLIED; identical-old-key replay returns exactly the original stale receipt. The whole case after the human change, the unique historical level-2 application, both canonical receipts and all React/source durable vectors must be unchanged by the attempted controls.
  • This checks two serialized human-before-intent cases and MDM's freshness/occurrence-history authority. It does not qualify the inverse human/claim interleaving, concurrent React worker claims, clone safety, recovery across missed deadlines, or v0.49 release. Pinned-producer CI #39428111533 passed on head f81ce2d. The separate recurrence addition requires fresh CI.

Concurrency gate — previous head a673e06 (passed)

  • Commit a673e06c96c1b5b3b8eb8c13aa9fd7da710ff8d4 extends the already-green post-human-reduction audit. A separate PostgreSQL connection holds a FOR UPDATE lock on the same MDM policy-case row, launches two distinct authenticated mdm_s1_runtime/pgreact_mdm_worker requests with different request keys, requires both sessions to be observed waiting on PostgreSQL Lock, then releases the row. Every resulting ACTION_DENIED / ESCALATION_LEVEL_ALREADY_APPLIED receipt and canonical body/digest is compared against the previously qualified denial vector. Apart from exactly two new immutable MDM denial receipts, the complete MDM case, React intent-request/attempt, core work/attempt and business/technical-source state must remain identical. This is MDM row-lock concurrency in the real React joint-stack fixture; it does not substitute for React claimed-episode race acceptance.
  • Pinned-producer CI #39421541299, PR core CI #39421541534, and legacy adapter #39421545532 passed on that exact older head. This includes the observed MDM case-row-lock concurrency proof. The new human-edit freshness test at f81ce2d requires its own checks.

New follow-up regression — 2026-10-20

  • On earlier React head bae5b29ed64d481f1e7973408950d651d51171b2, the exact MDM cross-key fixture now also replays the original APPLIED_CONTROL request key after the authorized human has lowered that level, requiring the original receipt and control (with no re-application). It continues to assert two distinct fresh denial keys and identical-key denial replay. Pinned-producer CI #39415705167 is queued; this additional replay assertion is not yet qualified.
  • On React 2404ae34, the audit additionally restores the legitimate worker login after the human edit and fresh-key denials, waits for another PostgreSQL postmaster/worker cycle, and asserts complete unchanged MDM/React durable state. This covers the current registered idle source only, not future typed rule declarations or all races. Pinned-producer CI #39415248547 is pending.
  • Legacy v0.47 adapter workflow no longer uses grep -q in a pipefail-sensitive Docker logs pipe, after the prior startup-probe failure #39406317004. The corrected workflow is running on earlier intermediate heads and pending on this head. It does not modify historical v0.47 evidence or source release tags.

Verified current-head status — 2026-10-20

All current-head React CI passed for 4919a71901cf66c70dcf9c442138c9c03b7e4be4, including pinned MDM producer registration, real idle ESCALATE delivery, and the new full-state quiescence assertion across later background-worker cycles. See pinned-producer CI and core CI. This is focused T49.01/T49.03/T49.07 evidence only; no cross-key race/restart/catch-up or joint release acceptance is implied. pg-trickle #1197 is a separate draft correctness fix, not a substituted producer for this passing React run. The PR remains draft with production activation disabled.

Summary

The direct MDM-source gate failed because pinned pg-trickle 0.108.3 drops JOIN … USING predicates while expanding nested views. Commit a901accef20ea912591586409f999708948c07d5 uses the equivalent explicit ON selected_case.case_key = policy_case.case_key predicate. The candidate still reads MDM's live due_at directly. The same complete gate passes locally; pushed-head CI is pending.

Admission accepts exactly 0.108.0 and MDM-pinned 0.108.3, retaining capability, scheduler, CDC, ratio and isolation guards. Producer source, worker permissions and gate assertions are unchanged.

Validation

  • Exact local source, artifact, configuration and complete vectors: PostgreSQL 18.6 ARM64, MDM 0.14.1 at 06471e647ee54a80973fb5ad6709830f9955dbff, pg-trickle 0.108.3 at d8bb8eb5d45c5c9982e173e88914037433a1ee05, React SQL 0.46.1. Local joint image sha256:3cdd0ae0ac2b15f42cabdfc08c41b758360807148f6a5399f05b9bff4066453a; unchanged compiled React base and SQL provenance are recorded separately.

  • The ordinary PostgreSQL-managed worker delivers typed level-2 ESCALATE 0.468853 seconds after a future deadline, without business-source edits or manual refresh/time calls. Complete cases, request/receipt bodies, source, work and attempt vectors pass, including callback failure/role reset and missing SET membership.

  • Exact supported/unsupported versions; missing/altered/disabled capabilities; malformed/duplicate discovery; missing API; incompatible scheduler/CDC/ratio/isolation; fresh/adjacent-upgrade callable/ACL parity all pass. Exact-target MAINTAIN refresh works without source SELECT, source-schema USAGE, owner membership, superuser or BYPASSRLS; unrelated/revoked grants and lifecycle operations are denied.

  • Unchanged MDM bootstrap/negative fixtures, v0.47 fixture/typed/live, v0.48 codec/live/security, missing-MDM admission and deterministic time boundaries pass. Documentation, source/evidence hashes and whitespace checks pass.

  • At previous head a901accef20ea912591586409f999708948c07d5, PR core CI #39383404472, pinned-producer CI #39383404414 and adapter CI #39383404459 passed. At current head 4919a71901cf66c70dcf9c442138c9c03b7e4be4, PR core CI #39388762584, push core CI #39388755996, pinned-producer integration #39388762423, and legacy adapter #39388762450 all passed. These checks qualify the exercised single-case runtime path, not full v0.49.0: Complete managed bounded escalation #11/v0.49.0: Qualify escalation safety, recovery and operating limits #12 acceptance.

  • The preceding direct-source failure and earlier technical-source CI evidence remain frozen and separately scoped.

  • T49.03/T49.07 focused passing regression (head 4919a71901cf66c70dcf9c442138c9c03b7e4be4): the real managed idle test waits for later background-worker heartbeat cycles and compares complete persisted MDM case/receipt, immutable React intent-request/attempt, completed work/attempt, and business/technical-source snapshots. Pinned-producer CI #39388762423 passed on the exact head. This proves post-delivery single-case idle quiescence, not different-key/human-reduction/concurrency/restart/backlog acceptance. No new automation or historical-evidence revision.

  • Upstream producer correctness: pg-trickle #1197 repairs genuine JOIN ... USING view expansion semantics but remains independent/draft with unpublished corrected binary and requested independent CODEOWNERS review. The existing React explicit-ON source still works on the original pinned producer; no substitute artifact or upstream merge is assumed.

  • Post-delivery restart/no-replay qualification (head 5c545d8): pinned-producer CI #39397414754, core CI #39397414503 and adapter CI #39397414602 passed. The test restarted the PostgreSQL postmaster after one applied control, observed multiple recovered worker cycles, and asserted no durable MDM/React case, receipt, request, attempt, work or source changes. This does not qualify downtime across new deadlines, coherent restore or clone activation; the later 66315d addition is separately pending.

  • Producer correctness merged, not adopted: pg-trickle #1197 was merged after explicit sole-maintainer authorization at source commit 04391eca2d53e548eafc84e82256df91eec01ea5; CI for its PR head was green, but source merge is not an independently packaged and qualified MDM producer. MDM #5 carries the source archive/checksum qualification requirement. This PR still tests the original MDM-pinned d8bb8eb producer using the explicit ON variant; no upstream artifact swap or privilege broadening is assumed.

  • Earlier human-reduction cross-key MDM authority fixture (66315de8813e9600964306ee842f9e871399617a): adds time/human-reduction-crosskey.sql and invokes it after real managed idle delivery, quiescence, post-delivery PostgreSQL restart/no-replay, and missing-SET-role safety. The test lowers the visible level through the authenticated MDM human API, submits two new distinct request keys through the authenticated MDM worker API, demands ACTION_DENIED / ESCALATION_LEVEL_ALREADY_APPLIED terminal receipts with complete canonical request bodies and unchanged full case, verifies the original applied receipt remains unique and checks same-key replay returns the identical denial receipt. This is MDM-side semantic safety within the real joint stack, not proof that React suppresses fresh event emissions after a human edit. Current-head pinned-producer CI is pending; no pass is claimed for the new fixture.

Verification update — 2026-10-20 (cross-key)

  • Qualified at 66315de8813e9600964306ee842f9e871399617a: pinned-producer CI #39406316972 and core CI #39406316996 passed. The pinned job explicitly executed human-reduction-crosskey.sql: an authenticated human reduced a level after automated level-2 application; two different fresh worker request keys received ACTION_DENIED / ESCALATION_LEVEL_ALREADY_APPLIED, complete receipt/request/case assertions passed, and a same-key retry replayed the original denial. This is MDM authority-path safety exercised in React's joint stack, not proof of React-side fresh-work suppression.
  • Legacy v0.47 adapter CI flake: run #39406317004 failed before SQL assertions; PostgreSQL initialization completed, but docker logs | grep -Fq under pipefail exited 1 due to an early-closed pipeline. Commit ee9abecc changes exactly those two readiness probes to consuming grep -F ... >/dev/null. The workflow still runs its original adapter checks and preserves historical release/tag sources. New CI pending.
  • Next runtime safety candidate: current head 2404ae34 restores the legitimate non-superuser managed-worker login after the human reduction and fresh-key denials, waits for a restarted postmaster and multiple live managed-worker cycles, and compares complete MDM case/receipt, React immutable intent/attempt, core work/attempt and source vectors. This is a more direct React convergence probe; pinned-producer CI #39415248547 is pending. No React-side no-op claim is made yet.

Remaining work

Current-head CI has passed. Resume focused #11 implementation/qualification for cross-key and human-change uniqueness, different occurrence recurrence, bounded next-level progression and catch-up, stale queued work, public diagnostics, concurrency, restart/restore and clone isolation. Preserve the qualifying idle-case test and its exact snapshots. The runtime proof does not complete #11.

Merge Danger

Keep PR #14 draft and #11 open. Production activation remains disabled. No #12 release qualification or pg-trickle #1168 closure is claimed; workload measurements and signed operating limits remain outstanding. No wildcard range, admission-guard removal, producer patch, source-copy workaround, stream ownership transfer or additional worker privilege is introduced. Published predecessor SQL and historical v0.48 evidence remain unchanged.

This was referenced Oct 19, 2026
Merged
Open
@grove ChatGPT Codex Connector

grove commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

#11 implementation handoff after MDM fixture qualification (2026-10-19)

MDM negative-receipt dependency is resolved. pg-mdm #19 merged at 06471e647ee54a80973fb5ad6709830f9955dbff. Final-head CI #39285572518 passed both test and packaged e2e; executable negative receipt vectors and expected outcomes are available on main. See #11 exact handoff.

The old CI failure is not a v0.49 feature failure. Frozen v0.48 release run #39266389941 rejected intentional changes to integrations/pg-mdm/sql/routing.sql and comparison.sql because of the archived v0.48 signed source hashes. Commit 214171b scopes the historical v0.48 release workflow to its tag and manual dispatch. It does not change any historical source hash, signed evidence, release verification step, or current CI; use ordinary CI and the pending v0.49 qualification to assess this PR.

Next coding scope in #11: Keep this PR's explicit-time evaluator, then identify and exercise the existing installed pgreact_api.author_deadline_rule, pgreact_api.run, pgreact_api.managed_cycle, and worker/claim path. Prove real idle-case liveness with a future due_at without business-source modification, deliver typed ESCALATE using the existing atomic MDM worker adapter, preserve the original due time, enforce at most one next level per sampled evaluation with capped admissions/catch-up, converge repeated polling to no-op, and add exact request/receipt/control and public diagnostics tests. The existing tests/integrations/pg-mdm/time/idle.sql is a feasibility/audit input, not passing v0.49 joint qualification. Recheck pause/replacement/human-edits/cross-key dedup, restart and clone isolation.

Status: PR #14 is still only preparatory (draft). Do not close #11 or enable escalation. Full release qualification remains #12; pg-trickle #1168 and signed numeric workload bounds remain open. New PR CI is running on the updated branch.

@grove grove changed the title Prepare explicit-time escalation evaluation for #11 Qualify pinned producer and prepare managed escalation for #11 Oct 20, 2026
@grove grove changed the title Qualify pinned producer and prepare managed escalation for #11 Qualify pinned producer: direct MDM-source gate blocked for #11 Oct 20, 2026
@grove grove changed the title Qualify pinned producer: direct MDM-source gate blocked for #11 Qualify pinned producer and direct MDM idle delivery for #11 Oct 20, 2026
@grove ChatGPT Codex Connector

grove commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Verified green CI and next implementation tranche (2026-10-20)

Latest exact head: 4919a71901cf66c70dcf9c442138c9c03b7e4be4. Pinned-producer #39388762423, PR core #39388762584, push core #39388755996, and legacy adapter #39388762450 all passed. The pinned-producer job executed tests/integrations/pg-mdm/time/audit.py with the added post-delivery full-state idle quiescence assertion. One authorized MDM idle APPLIED_CONTROL and no later state changes are now supported by CI evidence.

Next #11 tranche — preserve that runnable proof and its frozen prior evidence. Implement and qualify per-occurrence/level safety under two distinct request keys and a human lower-level action, genuine recurrence with a different MDM case key, both human-edit/claim interleavings, and stale/pause/replacement rechecks. Use real MDM receipt-history authority, not a React-only in-memory dedup shortcut. Then cover bounded cycle admission after downtime (no per-missed-poll replay), restart/restore/clone activation, and exposed public diagnostics with full expected/actual control/receipt vectors. Include exact final-artifact CI and do not simply convert old green evidence into new source qualification.

Scope: #11 remains open and this PR remains draft. Upstream producer #1197 has green PR-head CI but pending independent review, unpublished corrected binary and skipped broad E2E/benchmark lanes. The existing passing React producer is still its original MDM pin using explicit ON; no producer swap. #1168 throughput, #12 and signed numeric operating bounds remain separate.

@grove grove mentioned this pull request Oct 20, 2026
Open
8 tasks
grove added 13 commits October 20, 2026 12:12
@grove
test(mdm): record exact cross-key ESCALATE denials after human level …
336ba58 
…reduction
@grove
test(mdm): run human-reduction cross-key denial fixture after idle re…
66315de 
…covery
@grove
ci(v0.47): avoid pipefail SIGPIPE in PostgreSQL startup log probe
ee9abec
@grove
test(mdm): verify managed worker quiescence after human level reduction
2404ae3
@grove
test(mdm): replay original APPLIED receipt after human level reduction
bae5b29
@grove
test(mdm): verify simultaneous cross-key denials behind one case-row …
a673e06 
…lock
@grove
test(mdm): assert stale captured intent versus fresh retained-level d…
804de23 
…enial after human edit
@grove
test(mdm): exercise stale and current token outcomes after human change
f81ce2d
@grove
test(mdm): assert independent escalation on genuine recurring case oc…
3916dca 
…currence
@grove
test(mdm): enforce distinct recurrence case and typed timestamp equality
5019bd0
@grove
test(mdm): run genuine MDM recurrence isolation after stale-intent ch…
68918d7 
…ecks
@grove
test(mdm): strengthen recurrence preconditions and exact receipt digest
c837f8f
@grove
test(mdm): resolve recurrence from retained public case identities
d413f72
@grove grove mentioned this pull request Oct 21, 2026
Merged
@grove grove mentioned this pull request Oct 21, 2026
Merged
@grove grove mentioned this pull request Oct 21, 2026
Merged
@grove grove mentioned this pull request Oct 21, 2026
Merged
@grove ChatGPT Codex Connector

grove commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

2026-10-21 — backlog timeline integrated on feature branch

React PR #19 merged into the unmerged PR #14 feature branch, not main, as d70a73d. All final PR-head workflows passed on feb85b2d509ada846127999490cd28c356302e62: pinned producer #39677913209, MDM preview #39677913220, and core/adapter #39677913391. The earlier failed attempt remains historical evidence.

The two-case real outage test now retains sampled completion timestamps, per-case receipt sets, managed status, and an observed drain interval, asserting no duplicates, disappearing receipts, unrelated cases or incomplete recovery. It does not establish a hard per-cycle admission limit, mixed-backlog fairness, the signed 300-case throughput/recovery envelope or 60-cases/minute S1 qualification. Coherent restore/clone, operator sign-off and T49.06/T49.12 release gates remain open.

Post-merge head d70a73d has newly queued pinned producer #39681815208 and preview #39681815435, which must complete before exact merged-head qualification. Keep PR #14 draft, issues #8/#11/#12 open, and production inactive.

@grove grove mentioned this pull request Oct 21, 2026
Merged
@grove
Merge PR #20: prove one-item claim bound preserves pending backlog
6a55008 
test(v0.49): prove distinct one-item claims preserve pending backlog
This was referenced Oct 22, 2026
Draft
Open
Open
@grove grove mentioned this pull request Oct 22, 2026
Merged
@grove
Merge pull request #22 from trickle-labs/docs/v049-owner-approved-s1-…
6b80522 
…rate-floor

docs(v0.49): ratify approved 59/min acceptance floor, retain 60/min target
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@grove