From 2b96ea72fc3edc03297a01a489e03a9a17d62bff Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 15:34:23 +0000 Subject: [PATCH 01/28] =?UTF-8?q?docs(adr):=20ADR=200015=20=E2=80=94=20def?= =?UTF-8?q?erred-dispatch=20admission=20bound=20and=20fenced=20cross-insta?= =?UTF-8?q?nce=20coalescing=20(design=20for=20#44=20+=20#50)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/adr/0015-runtime-coordination.md | 214 ++++++++++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 docs/adr/0015-runtime-coordination.md diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md new file mode 100644 index 0000000..efb6d16 --- /dev/null +++ b/docs/adr/0015-runtime-coordination.md @@ -0,0 +1,214 @@ +# ADR 0015: Runtime Coordination — Admission Bound and Fenced Cross-Instance Coalescing + +## Status + +Proposed. This is the design for the deferred-dispatch admission bound (issue #44) and cross-instance coalescing (issue #50), decided together as one runtime-coordination track. It reshapes the force/steerability surface ADR 0012 proposed (`ForceReleaseLock` by key) and gives the staged burst/preemption branch (PR #53) its verdict. + +## Context + +ADR 0002 made `DispatchDeferred` the opt-in **Ack-Then-Work** mode: the adapter acknowledges after the synchronous prelude and the handler runs on the **Detached Work Context**, bounded by `DetachTimeout`. ADR 0012 expanded the **Concurrency Strategy** set; the reduced implementation on main ships `drop`, `queue`, `debounce`, and `concurrent` plus **Lock Scope** and universal lease-loss cancellation (a deferred holder whose **Lock Lease** is lost mid-run is cancelled within one refresh interval, `ThreadLockTTL/2`). Two costs were deliberately deferred: + +- **No admission bound (#44).** Every accepted routed event under `DispatchDeferred` retains a detached tail — goroutine, event payload, handler closure — until completion, supersession, or `DetachTimeout`. A unique-event flood grows retention linearly. Burst batch members would retain payloads without even holding a goroutine. +- **Per-instance coalescing (#50).** Queue supersession and debounce coalescing live in process memory (the pending-waiter registry). Instances sharing a production **Runtime State** each dispatch their own most-recent event; the **Thread Lock** serializes them, but nothing supersedes across instances, while the upstream semantic is global newest-wins. + +The burst strategy and the `OnLockConflict`/`ForceReleaseLock` preemption path were staged on PR #53 rather than merged, because nine review rounds on PR #38 kept finding P1s in exactly that machinery. Those findings were not random; they reduce to recurring failure classes, and this design's job is to preclude the classes structurally rather than patch instances of them: + +1. **Unbounded admission** — goroutines/payloads retained before any concurrency gate ([r3871060076](https://github.com/coder/chat/pull/38#discussion_r3871060076), [r3871214506](https://github.com/coder/chat/pull/38#discussion_r3871214506)). +2. **Process-local coordination with distributed semantics implied** — per-instance registries (waiters, cancels) that multi-instance deployments silently defeat ([r3871403308](https://github.com/coder/chat/pull/38#discussion_r3871403308), [r3871919968](https://github.com/coder/chat/pull/38#discussion_r3871919968)). +3. **Key-only force release** — deleting a lock by key alone races holder turnover and kills innocent successor leases ([r3871938987](https://github.com/coder/chat/pull/38#discussion_r3871938987)). +4. **Non-atomic ownership handoff** — choreography between lock acquisition, local registration maps, and supersession rechecks, each gap a time-of-check race ([r3872104899](https://github.com/coder/chat/pull/38#discussion_r3872104899), [r3872390326](https://github.com/coder/chat/pull/38#discussion_r3872390326)). +5. **Premature handover** — a preemptor proceeding while its victim still runs ([r3871040671](https://github.com/coder/chat/pull/38#discussion_r3871040671)). +6. **Lease-lifecycle divergence** — parallel refresh/cleanup loops re-deriving hardening the shared path already has ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204), [r3872390336](https://github.com/coder/chat/pull/38#discussion_r3872390336)). +7. **Temporal and budget skew** — shared batch deadlines starving late members; prelude stalls displacing newer waiters ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320), [r3872622515](https://github.com/coder/chat/pull/38#discussion_r3872622515)). + +One domain constraint is honored throughout: **Runtime State** is coordination state, not **Thread Application State** and not a message store (CONTEXT.md; the ADR 0009 **History Reader** storage rule). No design below places event payloads in State. + +## Decision + +Four pieces, decided together. Two are runtime-local (admission bound, burst shaping); two are optional **Runtime State** capabilities discovered by type assertion, per the ADR 0009 `HistoryReader` precedent — the required `State` interface does not change. + +### 1. Admission Bound (issue #44) + +Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot at acknowledgement; only routed deferred work holds it to the tail's end. + +Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to the platform's retry-inducing response (HTTP 429/503 equivalents; the mapping is adapter-owned). Because the delivery was never acknowledged and never marked, the platform's own retry redelivers it and is not deduped away — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away). + +Placement and validation: + +- The gate sits at the head of the prelude, before any **Runtime State** write, so a rejected delivery leaves no record. +- `MaxDetached` must be positive under `DispatchDeferred` (constructor validation, the `DetachTimeout` precedent); `DefaultRuntimeOptions` gains a default (1024). `DispatchSync` ignores it: synchronous dispatch runs inline on the webhook goroutine and is bounded by the HTTP server's own concurrency. +- Observability: a rejected delivery emits a new observation (`admission_rejected`) and closes its dispatch span with a new terminal outcome (`admission-rejected`). Rejections are never silent. + +Interaction with each strategy (under `DispatchDeferred`): + +| Strategy | What retains memory | What the bound caps | +|---|---|---| +| drop | running tails (one per scope) | total tails across scopes | +| queue | running tails + at most one parked waiter per scope | scope-cardinality floods | +| debounce | parked waiters (≤ 1 per scope) + running tails | scope-cardinality floods | +| concurrent | slot-waiters + at most `MaxConcurrent` running | the waiting line behind `MaxConcurrent` | +| burst (staged) | batch members (payload retained, no goroutine) + the running member | total retained members on the instance | + +### 2. Burst shaping (issue #44's scope extension) + +When burst ships (see the PR #53 outcome), batch growth is bounded twice: + +- **Globally** by `MaxDetached`: each member counts from admission to its member-run's terminal outcome. +- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) + +Two rules carried from the failure classes: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member (class 7) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder (class 6): one lease-lifecycle implementation, no parallel loop. Byte-based accounting is refused — see Non-goals. + +### 3. Cross-instance coalescing (issue #50): Waiter Fences on an optional `Coalescer` capability + +Queue supersession and debounce coalescing extend across instances through a per-scope, State-issued monotonic **Waiter Fence** — a fencing token that totally orders coalescing participants for a scope. New optional capability: + +```go +// Coalescer is an optional State capability providing per-scope fencing for +// cross-instance waiter supersession. Fences are strictly increasing per scope. +type Coalescer interface { + // NextFence allocates the next fence for the scope. + NextFence(ctx context.Context, scope string, ttl time.Duration) (uint64, error) + // RegisterWaiter records fence as the scope's newest waiter iff it is newer + // than the currently registered fence; registered=false means the caller is + // already superseded. + RegisterWaiter(ctx context.Context, scope string, fence uint64, ttl time.Duration) (bool, error) + // NewestWaiter reads the scope's currently registered fence, if any. + NewestWaiter(ctx context.Context, scope string) (uint64, bool, error) +} +``` + +Protocol — for `queue` and `debounce` only (burst is delivery-preserving and drop is first-wins; neither coalesces by fence): + +- **Allocate early.** A routed queue/debounce delivery allocates its fence right after the lock scope is derived, before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). +- **Register only on park.** Only a delivery that actually parks (a queue **Lock Conflict** waiter; every debounce waiter) registers its fence. Allocation alone supersedes no one — duplicates, unrouted events, and prelude failures can never displace a live waiter. `RegisterWaiter` returning false means a newer waiter already exists: the delivery skips immediately with the existing superseded observation, exactly like local supersession. +- **Check on dispatch.** A delivery that parked (or preempted — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. A delivery that never conflicted dispatches unconditionally — the in-flight turn is never retroactively superseded, matching local queue semantics. + +The guarantee split, stated honestly: + +- **Per-instance (unchanged, always):** most-recent-wins by admission order; superseded waiters exit promptly; skips are observable. +- **Cross-instance (capability present):** at most the newest registered waiter per coalesced group dispatches. "Newest" is defined by fence-allocation order at the shared State, which approximates arrival order — not platform send order — under delivery skew. +- **Capability absent:** exactly today's documented per-instance behavior — correct per instance, weaker globally, logged once at startup. Never a constructor error: per-instance coalescing is valid semantics, not a broken configuration. + +Failure modes accepted and documented, not hidden: + +- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its waiter (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL no shorter than the strategy's maximum park horizon; expiry degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). +- **Register unavailability degrades, never blocks.** A `Coalescer` error falls back to per-instance semantics for that dispatch, with an observation; events are never lost to a register outage — they serialize under the **Thread Lock** as today. + +Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (mutex + counter), **Redis State** (`INCR` + compare-greater script), **Postgres State** (per-scope row with a bigint fence + expiry), **NATS State** (KV writes whose per-key revisions provide the monotonic source). Third-party States keep compiling and keep today's semantics. + +### 4. Preemption on a fenced Lock Takeover (`LockForcer` reshaped; issue #50's scope extension) + +ADR 0012 proposed `ForceReleaseLock` by key. The #38 history shows why that shape cannot be made safe (classes 3–5): key-only identity cannot distinguish victim from successor, release-then-acquire leaves a gap a third party can enter, and the compensating local choreography (`inflightCancels`, `victimDone`) was the single largest P1 source. The reshaped surface: + +```go +// LockHolder identifies a Thread Lock holder for compare-and-take. Identity is +// an opaque per-lease identifier, stable across ExtendLock refreshes; it need +// not be — and should not be — a release-capable token. +type LockHolder struct { + Key string + Identity string +} + +// LockForcer is an optional State capability enabling preemption. +type LockForcer interface { + // ObserveLock reports the current holder of key, if held. + ObserveLock(ctx context.Context, key string) (LockHolder, bool, error) + // TakeLock atomically replaces the lock iff its current holder is still + // observed, returning a fresh Lock Lease for the caller. taken=false means + // the holder changed or the lock is free — never an error. + TakeLock(ctx context.Context, observed LockHolder, ttl time.Duration) (LockLease, bool, error) +} +``` + +- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. +- **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim stops via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted` within one refresh interval. +- **What preemption means, honestly:** preemption invalidates the victim's lease; it does not instantly kill the victim. Victim and preemptor may overlap for at most one refresh interval (`ThreadLockTTL/2`; 60s at defaults). `ThreadLockTTL` is the tuning knob: shorter TTL, shorter overlap, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). +- **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding) and requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently). +- **Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted. +- **Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here. + +### 5. Composition + +`Coalescer` and `LockForcer` are independent capabilities, and each alone is honest: coalescing without preemption gives global newest-wins with strategy-path conflict handling; preemption without coalescing gives fenced takeover with per-instance-only supersession (a stale preemptor may then dispatch its event — today's per-instance semantics, serialized as ever by the **Thread Lock**). Implementing both gives the full contract. The conformance suite tests each capability separately plus the composition. + +## Failure-mode disposition (the #38 anti-examples) + +| # | Class | Disposition in this design | +|---|---|---| +| 1 | Unbounded admission | Precluded: pre-ack `MaxDetached` gate; per-scope `MaxBurstBatch` seal-and-roll | +| 2 | Process-local coordination, distributed semantics | Precluded: cross-instance semantics are only claimed where a State capability backs them; absence degrades to documented per-instance behavior; preemption is single-path via State | +| 3 | Key-only force release | Precluded: `TakeLock` compare-and-swap on observed holder identity; no release→acquire gap | +| 4 | Non-atomic handoff/registration races | Precluded: no local preemption registries to race with State updates; cross-instance ordering is enforced by State atomics (fence CAS, lock CAS) | +| 5 | Premature handover | Transformed and bounded: no victim-drain handshake; overlap explicitly ≤ one refresh interval, tunable, documented | +| 6 | Lease-lifecycle divergence | Precluded by rule: all lock-holding paths (including burst) reuse the one refresh/cancel/outcome implementation | +| 7 | Temporal/budget skew | Precluded: early fence allocation pins cross-instance admission order; per-member execution budgets; no shared batch deadline | + +Accepted residuals, all observable: coalesced-turn loss on register-then-die (§3); up to one refresh interval of preemption overlap (§4); bounded spurious preemption by a superseded preemptor (§4); fence order approximating State-arrival order, not platform send order (§3). + +## Outcome for PR #53 (staged burst + preemption) + +**Verdict: close PR #53.** Neither half should merge in its current shape, and the halves should not travel together again. + +- **Burst: rewrite on this design, salvaging the concept and its tests.** `ConcurrencyBurst` and per-member fresh budgets survive. Required changes: members count against `MaxDetached`; `MaxBurstBatch` with seal-and-roll; the batch lock lifecycle must reuse the shared refresh/cancel machinery (the staged branch's separate refresh loop reproduced class 6 twice). Ship as its own PR after the admission bound lands. +- **Preemption: rewrite from scratch.** Rejected outright: key-only `ForceReleaseLock(ctx, key)` and all four backend implementations of it; the `inflightCancels` registry; `preemptLocalIfPending`/`victimDone`. Surviving with changed shape: `OnLockConflict` (same signature, now deferred-only and `LockForcer`-required at construction), `ErrPreempted`/`OutcomePreempted` (already on main), and the `LockForcer` capability name — carrying the `ObserveLock`/`TakeLock` contract above. Ship last, after the `Coalescer` fence exists, so the stale-preemptor check has something to read. + +## Non-goals + +This design explicitly refuses to promise: + +- **Exactly-once (or at-least-once) cross-instance dispatch.** Coalescing is at-most-newest per group; deferred dispatch's crash-loss contract (ADR 0002) is inherited, and supersede-then-crash loses the group's turn. +- **Cross-instance waiter takeover.** It would require event payloads in **Runtime State** — a message store, refused on the coordination-only rule. +- **Cross-instance burst batch merging.** Same refusal; per-instance batches serialized by the **Thread Lock** are the contract. +- **Byte-accounted admission.** The runtime cannot meaningfully measure retained `Raw` platform payloads plus handler closures; the bound is a count, and operators size it against their platform's payload ceiling. +- **Fleet-wide admission control.** `MaxDetached` protects one instance's memory; fleet capacity management belongs to the operator's front door. +- **Zero-overlap preemption.** A strict victim-drain handoff is exactly the class-4/5 churn machine; bounded overlap is the contract. +- **State watch/notify primitives.** Coalescing is check-at-dispatch; no backend is required to push notifications. + +## Consequences + +- Two new **Runtime Options** (`MaxDetached`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. +- Adapters gain one honesty duty: map `ErrAdmissionRejected` to the platform's retry-inducing response. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. +- The `State` interface itself does not change; both extensions are optional capabilities (ADR 0009 precedent). All four in-repo States implement both; the conformance suite grows capability sections (fence monotonicity, register-only-newer, TTL expiry, observe/take atomicity under contention, extend-churn non-defeat). +- Queue/debounce dispatch gains up to three State round-trips (allocate, register, check) when `Coalescer` is present — the cost of global newest-wins. Absent the capability, nothing new is paid. +- ADR 0012's proposed force surface is superseded by §4 (flagged per the domain docs' ADR-conflict rule). ADR 0012's status note and the CONTEXT.md glossary (**Admission Bound**, **Waiter Fence**, **Lock Takeover**) update when this ADR is accepted and implementation lands. +- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) `Coalescer` + backends + conformance; (3) queue/debounce fence integration — closes #50; (4) burst revival; (5) fenced preemption. + +## Alternatives Considered + +### Drop-with-observation at the admission cap + +Rejected. Acknowledging then discarding makes the platform 2xx a lie with no retry to correct it — the exact "new silent drop policy" #44 warned against. Observable-but-acked loss is still loss. + +### Block the webhook until capacity frees + +Rejected. Parking the webhook goroutine busts platform ack deadlines (Slack's 3s), converting overload into platform-side timeouts and retry storms — strictly worse backpressure than an honest retry signal. + +### Admission control in front of the webhook only (operator contract) + +Rejected as the whole answer. A load balancer cannot see detached-tail occupancy — the exhausted resource is invisible outside the runtime. Front-door rate limiting remains complementary. + +### Required `State` methods instead of optional capabilities + +Rejected. Growing the required interface breaks every third-party State for features many deployments never enable; the `HistoryReader` precedent (ADR 0009) established capability discovery with honest degradation. + +### Payload-bearing waiter registry in State (global coalescing with takeover) + +Rejected on the coordination-only rule: **Runtime State** would become a message store, with the size limits, retention, and privacy surface the contract deliberately excludes. The fence carries ordering, never content. + +### Key-only `ForceReleaseLock` (ADR 0012's proposed shape, PR #53's implementation) + +Rejected. Key-only identity cannot distinguish victim from successor (class 3), and release-then-acquire reopens the race even with identity. Compare-and-take on a single key moves the atomicity into a State primitive every backend can actually provide. + +### Local preemption fast-path registry (`inflightCancels` + `victimDone`) + +Rejected as correctness machinery. Two coordination systems — process maps and the State — must agree at every interleaving, and the #38 history is the catalog of ways they didn't. One path, through the State; local nudges may only ever be best-effort latency sugar. + +### Cross-instance debounce timer coordination (a global quiet period) + +Rejected. A globally-reset quiet period needs watch/notify primitives or polling loops in every parked waiter. Check-at-dispatch delivers the observable contract — only the newest dispatches — without new State primitives; the quiet period stays a per-instance approximation. + +### Single-phase fence allocation (allocate at registration time) + +Rejected. Registration order diverges from arrival order when preludes stall on State round-trips: a stale delivery could fence out a newer waiter (class 7, cross-instance). Allocating before the reorderable work pins the order; registering only on park keeps duplicates and failures from superseding anyone. + +Cross-references: ADR 0002 (deferred dispatch, crash-loss contract), ADR 0009 (optional-capability precedent), ADR 0012 (strategy set; force surface reshaped here), ADR 0014 (NATS State mechanics reused for fences and takeover); issues #44 and #50; the PR #38 review history and PR #53 (the staged branch this ADR disposes). From c4ecc31058ac6750987f495cd5b5713211f7eb09 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 15:49:05 +0000 Subject: [PATCH 02/28] docs(adr): honest preemption-overlap contract (cooperative cancellation) and explicit sync-dispatch serving-layer non-goal --- docs/adr/0015-runtime-coordination.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index efb6d16..a6485d8 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -36,7 +36,7 @@ Semantics at the cap: **reject-with-signal, before ack and before dedupe marking Placement and validation: - The gate sits at the head of the prelude, before any **Runtime State** write, so a rejected delivery leaves no record. -- `MaxDetached` must be positive under `DispatchDeferred` (constructor validation, the `DetachTimeout` precedent); `DefaultRuntimeOptions` gains a default (1024). `DispatchSync` ignores it: synchronous dispatch runs inline on the webhook goroutine and is bounded by the HTTP server's own concurrency. +- `MaxDetached` must be positive under `DispatchDeferred` (constructor validation, the `DetachTimeout` precedent); `DefaultRuntimeOptions` gains a default (1024). `DispatchSync` ignores it: a synchronous delivery's goroutine and payload belong to the HTTP request before dispatch begins, so a runtime-side cap cannot shed them — and `net/http` imposes no request-concurrency limit by itself. Bounding synchronous serving is therefore an explicit operator contract at the HTTP layer (a request/connection limiter in front of the **Webhook Handler**), stated in the option's GoDoc rather than assumed — see Non-goals. - Observability: a rejected delivery emits a new observation (`admission_rejected`) and closes its dispatch span with a new terminal outcome (`admission-rejected`). Rejections are never silent. Interaction with each strategy (under `DispatchDeferred`): @@ -121,8 +121,8 @@ type LockForcer interface { ``` - **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. -- **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim stops via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted` within one refresh interval. -- **What preemption means, honestly:** preemption invalidates the victim's lease; it does not instantly kill the victim. Victim and preemptor may overlap for at most one refresh interval (`ThreadLockTTL/2`; 60s at defaults). `ThreadLockTTL` is the tuning knob: shorter TTL, shorter overlap, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). +- **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted` within one refresh interval. +- **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval (`ThreadLockTTL/2`; 60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). - **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding) and requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently). - **Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted. - **Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here. @@ -139,11 +139,11 @@ type LockForcer interface { | 2 | Process-local coordination, distributed semantics | Precluded: cross-instance semantics are only claimed where a State capability backs them; absence degrades to documented per-instance behavior; preemption is single-path via State | | 3 | Key-only force release | Precluded: `TakeLock` compare-and-swap on observed holder identity; no release→acquire gap | | 4 | Non-atomic handoff/registration races | Precluded: no local preemption registries to race with State updates; cross-instance ordering is enforced by State atomics (fence CAS, lock CAS) | -| 5 | Premature handover | Transformed and bounded: no victim-drain handshake; overlap explicitly ≤ one refresh interval, tunable, documented | +| 5 | Premature handover | Transformed and documented: no victim-drain handshake; the victim's context is cancelled within one refresh interval, termination is cooperative — overlap lasts the signal latency plus the victim's cancellation latency (unbounded only for handlers that ignore their context) | | 6 | Lease-lifecycle divergence | Precluded by rule: all lock-holding paths (including burst) reuse the one refresh/cancel/outcome implementation | | 7 | Temporal/budget skew | Precluded: early fence allocation pins cross-instance admission order; per-member execution budgets; no shared batch deadline | -Accepted residuals, all observable: coalesced-turn loss on register-then-die (§3); up to one refresh interval of preemption overlap (§4); bounded spurious preemption by a superseded preemptor (§4); fence order approximating State-arrival order, not platform send order (§3). +Accepted residuals, all observable: coalesced-turn loss on register-then-die (§3); preemption overlap of one refresh interval plus the victim's cooperative cancellation latency (§4); bounded spurious preemption by a superseded preemptor (§4); fence order approximating State-arrival order, not platform send order (§3). ## Outcome for PR #53 (staged burst + preemption) @@ -161,7 +161,8 @@ This design explicitly refuses to promise: - **Cross-instance burst batch merging.** Same refusal; per-instance batches serialized by the **Thread Lock** are the contract. - **Byte-accounted admission.** The runtime cannot meaningfully measure retained `Raw` platform payloads plus handler closures; the bound is a count, and operators size it against their platform's payload ceiling. - **Fleet-wide admission control.** `MaxDetached` protects one instance's memory; fleet capacity management belongs to the operator's front door. -- **Zero-overlap preemption.** A strict victim-drain handoff is exactly the class-4/5 churn machine; bounded overlap is the contract. +- **Zero-overlap — or even hard-bounded — preemption.** A strict victim-drain handoff is exactly the class-4/5 churn machine, and Go cannot forcibly stop a handler regardless: the contract is prompt cancellation and cooperative termination, nothing stronger. +- **Admission control for synchronous dispatch.** Under `DispatchSync` the goroutine and payload exist at the HTTP layer before the runtime sees the delivery; a runtime cap cannot shed that load, and `net/http` imposes no request-concurrency limit of its own. Bounding synchronous serving is an explicit serving-layer operator contract (request limiting in front of the **Webhook Handler**), not a runtime promise. - **State watch/notify primitives.** Coalescing is check-at-dispatch; no backend is required to push notifications. ## Consequences From ac06901c3d32670aef5e74eff912793ce66f8ac5 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 16:03:36 +0000 Subject: [PATCH 03/28] =?UTF-8?q?docs(adr):=20close=20coalescing=20protoco?= =?UTF-8?q?l=20holes=20=E2=80=94=20global=20fence=20sequence,=20register-a?= =?UTF-8?q?ll-routed,=20deferred-only=20fencing,=20abandonment=20vs=20degr?= =?UTF-8?q?adation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/adr/0015-runtime-coordination.md | 41 ++++++++++++++++----------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index a6485d8..8e23c2e 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -60,14 +60,16 @@ Two rules carried from the failure classes: each batch member runs with its own ### 3. Cross-instance coalescing (issue #50): Waiter Fences on an optional `Coalescer` capability -Queue supersession and debounce coalescing extend across instances through a per-scope, State-issued monotonic **Waiter Fence** — a fencing token that totally orders coalescing participants for a scope. New optional capability: +Queue supersession and debounce coalescing extend across instances through a State-issued monotonic **Waiter Fence** — a fencing token that totally orders coalescing participants. New optional capability: ```go -// Coalescer is an optional State capability providing per-scope fencing for -// cross-instance waiter supersession. Fences are strictly increasing per scope. +// Coalescer is an optional State capability providing fenced cross-instance +// waiter supersession. type Coalescer interface { - // NextFence allocates the next fence for the scope. - NextFence(ctx context.Context, scope string, ttl time.Duration) (uint64, error) + // NextFence allocates the next fence from one non-resetting monotonic + // sequence shared by all scopes. Allocator state is O(1) and permanent; + // fences are never reissued. + NextFence(ctx context.Context) (uint64, error) // RegisterWaiter records fence as the scope's newest waiter iff it is newer // than the currently registered fence; registered=false means the caller is // already superseded. @@ -77,24 +79,26 @@ type Coalescer interface { } ``` -Protocol — for `queue` and `debounce` only (burst is delivery-preserving and drop is first-wins; neither coalesces by fence): +Fences come from **one global, non-resetting sequence**, not per-scope counters: a per-scope counter would need a TTL to avoid unbounded per-scope State growth, and an expired-then-reset counter would let an old delivery's high fence outrank a fresh low one, reversing newest-wins. A global sequence keeps allocator state O(1) and permanent while per-scope *register* entries expire freely — an expired register only ever degrades to per-instance semantics, never reorders. -- **Allocate early.** A routed queue/debounce delivery allocates its fence right after the lock scope is derived, before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). -- **Register only on park.** Only a delivery that actually parks (a queue **Lock Conflict** waiter; every debounce waiter) registers its fence. Allocation alone supersedes no one — duplicates, unrouted events, and prelude failures can never displace a live waiter. `RegisterWaiter` returning false means a newer waiter already exists: the delivery skips immediately with the existing superseded observation, exactly like local supersession. -- **Check on dispatch.** A delivery that parked (or preempted — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. A delivery that never conflicted dispatches unconditionally — the in-flight turn is never retroactively superseded, matching local queue semantics. +Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. + +- **Allocate early.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). +- **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. +- **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. The guarantee split, stated honestly: - **Per-instance (unchanged, always):** most-recent-wins by admission order; superseded waiters exit promptly; skips are observable. -- **Cross-instance (capability present):** at most the newest registered waiter per coalesced group dispatches. "Newest" is defined by fence-allocation order at the shared State, which approximates arrival order — not platform send order — under delivery skew. -- **Capability absent:** exactly today's documented per-instance behavior — correct per instance, weaker globally, logged once at startup. Never a constructor error: per-instance coalescing is valid semantics, not a broken configuration. +- **Cross-instance (capability present, deferred dispatch):** among registered deliveries that have not yet claimed their turn, at most the newest dispatches. "Newest" is defined by fence-allocation order at the shared State, which approximates arrival order — not platform send order — under delivery skew. +- **Capability absent (or sync dispatch):** exactly today's documented per-instance behavior — correct per instance, weaker globally, logged once at startup. Never a constructor error: per-instance coalescing is valid semantics, not a broken configuration. Failure modes accepted and documented, not hidden: -- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its waiter (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL no shorter than the strategy's maximum park horizon; expiry degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). -- **Register unavailability degrades, never blocks.** A `Coalescer` error falls back to per-instance semantics for that dispatch, with an observation; events are never lost to a register outage — they serialize under the **Thread Lock** as today. +- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL no shorter than the deferred park horizon (`DetachTimeout`); expiry degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). +- **Backend unavailability degrades; the delivery's own cancellation does not.** A `Coalescer` backend error falls back to per-instance semantics for that dispatch, with an observation; events are never lost to a register outage — they serialize under the **Thread Lock** as today. Cancellation or deadline expiry of the delivery's own context is *not* a degradation case: it is the existing abandonment path (exit without running), so a delivery whose execution budget ended at the dispatch-time check can never "fall back" into running the handler past its budget. -Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (mutex + counter), **Redis State** (`INCR` + compare-greater script), **Postgres State** (per-scope row with a bigint fence + expiry), **NATS State** (KV writes whose per-key revisions provide the monotonic source). Third-party States keep compiling and keep today's semantics. +Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key + register keys). Third-party States keep compiling and keep today's semantics. ### 4. Preemption on a fenced Lock Takeover (`LockForcer` reshaped; issue #50's scope extension) @@ -163,6 +167,7 @@ This design explicitly refuses to promise: - **Fleet-wide admission control.** `MaxDetached` protects one instance's memory; fleet capacity management belongs to the operator's front door. - **Zero-overlap — or even hard-bounded — preemption.** A strict victim-drain handoff is exactly the class-4/5 churn machine, and Go cannot forcibly stop a handler regardless: the contract is prompt cancellation and cooperative termination, nothing stronger. - **Admission control for synchronous dispatch.** Under `DispatchSync` the goroutine and payload exist at the HTTP layer before the runtime sees the delivery; a runtime cap cannot shed that load, and `net/http` imposes no request-concurrency limit of its own. Bounding synchronous serving is an explicit serving-layer operator contract (request limiting in front of the **Webhook Handler**), not a runtime promise. +- **Cross-instance coalescing for synchronous dispatch.** A synchronous queue park is bounded only by the caller's request context, which need not carry a deadline, so no sound registration TTL exists for it. Sync queue keeps per-instance semantics; cross-instance coalescing pairs with `DispatchDeferred`, whose parks `DetachTimeout` bounds. - **State watch/notify primitives.** Coalescing is check-at-dispatch; no backend is required to push notifications. ## Consequences @@ -170,7 +175,7 @@ This design explicitly refuses to promise: - Two new **Runtime Options** (`MaxDetached`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. - Adapters gain one honesty duty: map `ErrAdmissionRejected` to the platform's retry-inducing response. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. - The `State` interface itself does not change; both extensions are optional capabilities (ADR 0009 precedent). All four in-repo States implement both; the conformance suite grows capability sections (fence monotonicity, register-only-newer, TTL expiry, observe/take atomicity under contention, extend-churn non-defeat). -- Queue/debounce dispatch gains up to three State round-trips (allocate, register, check) when `Coalescer` is present — the cost of global newest-wins. Absent the capability, nothing new is paid. +- Queue/debounce dispatch under `DispatchDeferred` gains up to three State round-trips (allocate, register, check) when `Coalescer` is present — the cost of global newest-wins. Absent the capability, or under sync dispatch, nothing new is paid. - ADR 0012's proposed force surface is superseded by §4 (flagged per the domain docs' ADR-conflict rule). ADR 0012's status note and the CONTEXT.md glossary (**Admission Bound**, **Waiter Fence**, **Lock Takeover**) update when this ADR is accepted and implementation lands. - Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) `Coalescer` + backends + conformance; (3) queue/debounce fence integration — closes #50; (4) burst revival; (5) fenced preemption. @@ -210,6 +215,10 @@ Rejected. A globally-reset quiet period needs watch/notify primitives or polling ### Single-phase fence allocation (allocate at registration time) -Rejected. Registration order diverges from arrival order when preludes stall on State round-trips: a stale delivery could fence out a newer waiter (class 7, cross-instance). Allocating before the reorderable work pins the order; registering only on park keeps duplicates and failures from superseding anyone. +Rejected. Registration order diverges from arrival order when preludes stall on State round-trips: a stale delivery could fence out a newer waiter (class 7, cross-instance). Allocating before the reorderable work pins the order; registering only routed deliveries keeps duplicates and failures from superseding anyone. + +### Per-scope fence counters with TTL + +Rejected. A per-scope counter must either live forever — unique-scope traffic then grows Memory/Redis/Postgres coordination state permanently — or expire, and an expired-then-reset counter lets an old delivery's high fence outrank a fresh low one, reversing newest-wins exactly when a scope goes quiet. One global non-resetting sequence costs O(1) permanent state, preserves strict ordering across expiry of the per-scope register entries, and its per-scope subsequence is still strictly increasing. Cross-references: ADR 0002 (deferred dispatch, crash-loss contract), ADR 0009 (optional-capability precedent), ADR 0012 (strategy set; force surface reshaped here), ADR 0014 (NATS State mechanics reused for fences and takeover); issues #44 and #50; the PR #38 review history and PR #53 (the staged branch this ADR disposes). From aba6b2aa3687bb74ef620e9d625afcfb3523d4e6 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 16:18:41 +0000 Subject: [PATCH 04/28] docs(adr): commit-point degradation rule, admission-anchored register TTL, NATS uniform-TTL provision, reject preemption under concurrent --- docs/adr/0015-runtime-coordination.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 8e23c2e..90f24cd 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -95,10 +95,10 @@ The guarantee split, stated honestly: Failure modes accepted and documented, not hidden: -- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL no shorter than the deferred park horizon (`DetachTimeout`); expiry degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). -- **Backend unavailability degrades; the delivery's own cancellation does not.** A `Coalescer` backend error falls back to per-instance semantics for that dispatch, with an observation; events are never lost to a register outage — they serialize under the **Thread Lock** as today. Cancellation or deadline expiry of the delivery's own context is *not* a degradation case: it is the existing abandonment path (exit without running), so a delivery whose execution budget ended at the dispatch-time check can never "fall back" into running the handler past its budget. +- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL sized to the delivery's full possible park: `DetachTimeout` **plus** the bounded interval between registration (which happens in the request-context prelude) and tail start, with a safety margin — `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does. Expiry degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). +- **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. -Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key + register keys). Third-party States keep compiling and keep today's semantics. +Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register TTL must cover the fleet's maximum park horizon, and mixed-`DetachTimeout` fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. ### 4. Preemption on a fenced Lock Takeover (`LockForcer` reshaped; issue #50's scope extension) @@ -127,7 +127,7 @@ type LockForcer interface { - **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. - **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted` within one refresh interval. - **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval (`ThreadLockTTL/2`; 60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). -- **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding) and requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently). +- **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). - **Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted. - **Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here. From 1fbddf80e167bdcf80932618126743ed1df50955 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 16:29:23 +0000 Subject: [PATCH 05/28] docs(adr): bounded allocation-to-registration window with TTL arithmetic covering it --- docs/adr/0015-runtime-coordination.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 90f24cd..cc55fba 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -83,7 +83,7 @@ Fences come from **one global, non-resetting sequence**, not per-scope counters: Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. -- **Allocate early.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). +- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded window from its allocation, enforced on a local monotonic clock: past the window, a marked delivery proceeds per-instance (unfenced, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. - **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. - **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. @@ -95,7 +95,7 @@ The guarantee split, stated honestly: Failure modes accepted and documented, not hidden: -- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL sized to the delivery's full possible park: `DetachTimeout` **plus** the bounded interval between registration (which happens in the request-context prelude) and tail start, with a safety margin — `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does. Expiry degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). +- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL no shorter than the registration window **plus** the delivery's full possible park (`DetachTimeout` plus the bounded interval between registration — which happens in the request-context prelude — and tail start), with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). - **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register TTL must cover the fleet's maximum park horizon, and mixed-`DetachTimeout` fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. @@ -147,7 +147,7 @@ type LockForcer interface { | 6 | Lease-lifecycle divergence | Precluded by rule: all lock-holding paths (including burst) reuse the one refresh/cancel/outcome implementation | | 7 | Temporal/budget skew | Precluded: early fence allocation pins cross-instance admission order; per-member execution budgets; no shared batch deadline | -Accepted residuals, all observable: coalesced-turn loss on register-then-die (§3); preemption overlap of one refresh interval plus the victim's cooperative cancellation latency (§4); bounded spurious preemption by a superseded preemptor (§4); fence order approximating State-arrival order, not platform send order (§3). +Accepted residuals, all observable: coalesced-turn loss on register-then-die (§3); preemption overlap of one refresh interval plus the victim's cooperative cancellation latency (§4); bounded spurious preemption by a superseded preemptor (§4); fence order approximating State-arrival order, not platform send order (§3); per-instance degradation for deliveries whose prelude stalls past the registration window (§3). ## Outcome for PR #53 (staged burst + preemption) From a9b18a5d2b6a1316dc79cea39f1fb8e8889fe33b Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 16:42:50 +0000 Subject: [PATCH 06/28] docs(adr): concrete fence timing bounds, uniform-fleet caveat, LockForcer commit-point rule, bounded refresh calls, burst requires deferred --- docs/adr/0015-runtime-coordination.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index cc55fba..99ae15e 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -51,7 +51,7 @@ Interaction with each strategy (under `DispatchDeferred`): ### 2. Burst shaping (issue #44's scope extension) -When burst ships (see the PR #53 outcome), batch growth is bounded twice: +Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below — members counted by `MaxDetached`, fresh per-member budgets — are defined only for deferred dispatch. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission to its member-run's terminal outcome. - **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) @@ -83,22 +83,22 @@ Fences come from **one global, non-resetting sequence**, not per-scope counters: Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. -- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded window from its allocation, enforced on a local monotonic clock: past the window, a marked delivery proceeds per-instance (unfenced, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. +- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window from its allocation — one `ThreadLockTTL`, enforced by the runtime on a local monotonic clock — and the runtime likewise requires the dispatch-time check to begin within one `ThreadLockTTL` of registration (both bounds are runtime-enforced, not assumed). Past either bound, a marked delivery proceeds per-instance (unfenced or degraded, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. - **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. - **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. The guarantee split, stated honestly: - **Per-instance (unchanged, always):** most-recent-wins by admission order; superseded waiters exit promptly; skips are observable. -- **Cross-instance (capability present, deferred dispatch):** among registered deliveries that have not yet claimed their turn, at most the newest dispatches. "Newest" is defined by fence-allocation order at the shared State, which approximates arrival order — not platform send order — under delivery skew. +- **Cross-instance (capability present, deferred dispatch, uniform fleet):** among registered deliveries that have not yet claimed their turn, at most the newest dispatches. "Newest" is defined by fence-allocation order at the shared State, which approximates arrival order — not platform send order — under delivery skew. The guarantee assumes every instance sharing the State participates: during a rolling upgrade, deliveries handled by non-participating instances (older runtimes, or runtimes whose State lacks the capability) dispatch unfenced and are invisible to fenced ordering — cross-instance newest-wins is suspended for exactly those deliveries and resumes once the fleet is uniform; fenced deliveries still order among themselves. A capability/version gate that could enforce fleet uniformity from within the runtime is refused (it would need fleet membership in State); uniformity is an operator rollout contract, documented, like the preemption mixed-fleet rule in §4. - **Capability absent (or sync dispatch):** exactly today's documented per-instance behavior — correct per instance, weaker globally, logged once at startup. Never a constructor error: per-instance coalescing is valid semantics, not a broken configuration. Failure modes accepted and documented, not hidden: -- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL no shorter than the registration window **plus** the delivery's full possible park (`DetachTimeout` plus the bounded interval between registration — which happens in the request-context prelude — and tail start), with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). +- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the tail-start bound plus `DetachTimeout` — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both leading intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). - **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. -Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register TTL must cover the fleet's maximum park horizon, and mixed-`DetachTimeout` fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. +Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. ### 4. Preemption on a fenced Lock Takeover (`LockForcer` reshaped; issue #50's scope extension) @@ -124,9 +124,9 @@ type LockForcer interface { } ``` -- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. -- **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted` within one refresh interval. -- **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval (`ThreadLockTTL/2`; 60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). +- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. `ObserveLock`/`TakeLock` *failures* follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. +- **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded, with a refresh that cannot complete within its bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound. +- **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (`ThreadLockTTL/2`; ~60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). - **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). - **Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted. - **Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here. @@ -143,7 +143,7 @@ type LockForcer interface { | 2 | Process-local coordination, distributed semantics | Precluded: cross-instance semantics are only claimed where a State capability backs them; absence degrades to documented per-instance behavior; preemption is single-path via State | | 3 | Key-only force release | Precluded: `TakeLock` compare-and-swap on observed holder identity; no release→acquire gap | | 4 | Non-atomic handoff/registration races | Precluded: no local preemption registries to race with State updates; cross-instance ordering is enforced by State atomics (fence CAS, lock CAS) | -| 5 | Premature handover | Transformed and documented: no victim-drain handshake; the victim's context is cancelled within one refresh interval, termination is cooperative — overlap lasts the signal latency plus the victim's cancellation latency (unbounded only for handlers that ignore their context) | +| 5 | Premature handover | Transformed and documented: no victim-drain handshake; the victim's context is cancelled within one refresh interval plus the bounded refresh call, termination is cooperative — overlap lasts the signal latency plus the victim's cancellation latency (unbounded only for handlers that ignore their context) | | 6 | Lease-lifecycle divergence | Precluded by rule: all lock-holding paths (including burst) reuse the one refresh/cancel/outcome implementation | | 7 | Temporal/budget skew | Precluded: early fence allocation pins cross-instance admission order; per-member execution budgets; no shared batch deadline | @@ -177,7 +177,7 @@ This design explicitly refuses to promise: - The `State` interface itself does not change; both extensions are optional capabilities (ADR 0009 precedent). All four in-repo States implement both; the conformance suite grows capability sections (fence monotonicity, register-only-newer, TTL expiry, observe/take atomicity under contention, extend-churn non-defeat). - Queue/debounce dispatch under `DispatchDeferred` gains up to three State round-trips (allocate, register, check) when `Coalescer` is present — the cost of global newest-wins. Absent the capability, or under sync dispatch, nothing new is paid. - ADR 0012's proposed force surface is superseded by §4 (flagged per the domain docs' ADR-conflict rule). ADR 0012's status note and the CONTEXT.md glossary (**Admission Bound**, **Waiter Fence**, **Lock Takeover**) update when this ADR is accepted and implementation lands. -- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) `Coalescer` + backends + conformance; (3) queue/debounce fence integration — closes #50; (4) burst revival; (5) fenced preemption. +- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) `Coalescer` + backends + conformance; (3) queue/debounce fence integration — closes #50; (4) burst revival; (5) fenced preemption, including the independently time-bounded lease-refresh calls. ## Alternatives Considered From a8afa206f63cfadde4fe2732fb6013961b906dc1 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 16:59:03 +0000 Subject: [PATCH 07/28] docs(adr): full-park check horizon, fence as single ordering source, idempotent takeover reconciliation, shape-aware admission rejection --- docs/adr/0015-runtime-coordination.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 99ae15e..fba8131 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -31,7 +31,10 @@ Four pieces, decided together. Two are runtime-local (admission bound, burst sha Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot at acknowledgement; only routed deferred work holds it to the tail's end. -Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to the platform's retry-inducing response (HTTP 429/503 equivalents; the mapping is adapter-owned). Because the delivery was never acknowledged and never marked, the platform's own retry redelivers it and is not deduped away — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away). +Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to a shape-aware response (adapter-owned). Because the delivery was never acknowledged and never marked, no dedupe record blocks a retry — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away). The mapping is shape-aware because platforms do not retry every webhook shape: + +- **Platform-retried deliveries** (e.g. Slack Events API callbacks): the adapter returns the retry-inducing response (HTTP 429/503 equivalents) and the platform's own redelivery covers the event. +- **Direct user invocations that the platform does not redeliver** (e.g. Slack slash commands and interactivity — the in-repo Slack normalization records that these carry no retry headers): a bare 429 would turn a click into a silent permanent failure, so the adapter must answer with a *truthful busy response* to the user (a visible "busy, try again" acknowledgement, observed as rejected) — never a silent failure, never a fake success. The user's own retry is honored because no dedupe record exists. Placement and validation: @@ -83,7 +86,8 @@ Fences come from **one global, non-resetting sequence**, not per-scope counters: Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. -- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window from its allocation — one `ThreadLockTTL`, enforced by the runtime on a local monotonic clock — and the runtime likewise requires the dispatch-time check to begin within one `ThreadLockTTL` of registration (both bounds are runtime-enforced, not assumed). Past either bound, a marked delivery proceeds per-instance (unfenced or degraded, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. +- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window from its allocation — one `ThreadLockTTL`, enforced by the runtime on a local monotonic clock — and the runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed. Past either bound, a marked delivery proceeds per-instance (unfenced or degraded, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. +- **One ordering source.** When the capability is present, the **Waiter Fence** *is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders only unfenced deliveries — capability absent, sync dispatch, or degraded — and a mix of fenced and degraded deliveries follows today's per-instance rules with per-instance honesty. - **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. - **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. @@ -95,7 +99,7 @@ The guarantee split, stated honestly: Failure modes accepted and documented, not hidden: -- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the tail-start bound plus `DetachTimeout` — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both leading intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). +- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). - **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. @@ -115,16 +119,19 @@ type LockHolder struct { // LockForcer is an optional State capability enabling preemption. type LockForcer interface { - // ObserveLock reports the current holder of key, if held. + // ObserveLock reports the current holder of key, if held. A holder + // installed via TakeLock must be recognizable to its caller through + // ObserveLock (identity equal to, or a specified derivation of, the + // caller-generated replacement token). ObserveLock(ctx context.Context, key string) (LockHolder, bool, error) // TakeLock atomically replaces the lock iff its current holder is still - // observed, returning a fresh Lock Lease for the caller. taken=false means - // the holder changed or the lock is free — never an error. - TakeLock(ctx context.Context, observed LockHolder, ttl time.Duration) (LockLease, bool, error) + // observed, installing the caller-generated replacement lease. taken=false + // means the holder changed or the lock is free — never an error. + TakeLock(ctx context.Context, observed LockHolder, replacement LockLease, ttl time.Duration) (bool, error) } ``` -- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. `ObserveLock`/`TakeLock` *failures* follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. +- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. Only a confirmed non-commit is classified as a failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. - **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded, with a refresh that cannot complete within its bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound. - **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (`ThreadLockTTL/2`; ~60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). - **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). @@ -173,7 +180,7 @@ This design explicitly refuses to promise: ## Consequences - Two new **Runtime Options** (`MaxDetached`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. -- Adapters gain one honesty duty: map `ErrAdmissionRejected` to the platform's retry-inducing response. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. +- Adapters gain one honesty duty: map `ErrAdmissionRejected` shape-awarely — a retry-inducing response for platform-redelivered shapes, a truthful busy response for direct invocations the platform does not redeliver. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. - The `State` interface itself does not change; both extensions are optional capabilities (ADR 0009 precedent). All four in-repo States implement both; the conformance suite grows capability sections (fence monotonicity, register-only-newer, TTL expiry, observe/take atomicity under contention, extend-churn non-defeat). - Queue/debounce dispatch under `DispatchDeferred` gains up to three State round-trips (allocate, register, check) when `Coalescer` is present — the cost of global newest-wins. Absent the capability, or under sync dispatch, nothing new is paid. - ADR 0012's proposed force surface is superseded by §4 (flagged per the domain docs' ADR-conflict rule). ADR 0012's status note and the CONTEXT.md glossary (**Admission Bound**, **Waiter Fence**, **Lock Takeover**) update when this ADR is accepted and implementation lands. From 252b84f226cba3f07f524a2b045597c92e1e267f Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 17:10:32 +0000 Subject: [PATCH 08/28] docs(adr): bounded idempotent takeover reconciliation; fleet-uniform timing for TTL arithmetic across all backends --- docs/adr/0015-runtime-coordination.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index fba8131..7d24c55 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -99,7 +99,7 @@ The guarantee split, stated honestly: Failure modes accepted and documented, not hidden: -- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). +- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. The arithmetic assumes fleet-uniform `ThreadLockTTL` and `DetachTimeout` — the same uniform-fleet operator contract as capability participation: a short-timeout instance's registration could otherwise expire while a long-timeout instance's older fence is still within its horizon, reopening the empty-register reordering. Heterogeneous fleets must therefore derive every backend's registration TTL from fleet-wide maxima — for the per-call-TTL backends (Memory, Redis, Postgres) as much as for NATS's bucket TTL. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). - **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. @@ -131,7 +131,7 @@ type LockForcer interface { } ``` -- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. Only a confirmed non-commit is classified as a failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. +- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The reconciliation read is itself fallible, so it must be retried — reads are idempotent — on an independently bounded context until the replacement is confirmed present or definitively absent, or a bounded reconciliation budget expires; an outcome still inconclusive after the budget falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. - **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded, with a refresh that cannot complete within its bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound. - **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (`ThreadLockTTL/2`; ~60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). - **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). From 2725f90986c83d45b3b729ae56e02957ea5600e0 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 17:23:15 +0000 Subject: [PATCH 09/28] docs(adr): scope-wide degradation ordering, pre-RPC window anchor, expired register-entry purge --- docs/adr/0015-runtime-coordination.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 7d24c55..6c64fdf 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -86,8 +86,8 @@ Fences come from **one global, non-resetting sequence**, not per-scope counters: Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. -- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window from its allocation — one `ThreadLockTTL`, enforced by the runtime on a local monotonic clock — and the runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed. Past either bound, a marked delivery proceeds per-instance (unfenced or degraded, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. -- **One ordering source.** When the capability is present, the **Waiter Fence** *is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders only unfenced deliveries — capability absent, sync dispatch, or degraded — and a mix of fenced and degraded deliveries follows today's per-instance rules with per-instance honesty. +- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed. Past either bound, a marked delivery proceeds per-instance (unfenced or degraded, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. +- **One ordering source.** When the capability is present, the **Waiter Fence** *is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders deliveries only where fences are absent altogether (capability absent, sync dispatch). Degradation is therefore **scope-wide, not per-delivery**: a `Coalescer` failure degrades that delivery's scope on that instance to local admission-sequence ordering — local supersession and the dispatch-time check are suspended for the scope's currently-admitted deliveries until they drain — so fenced and unfenced deliveries are never ordered against each other by two different sources (mixed populations admit no consistent order). A degraded scope's earlier registrations still stand for other instances until they expire, which is the already-documented weaker-globally degradation. - **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. - **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. @@ -102,7 +102,7 @@ Failure modes accepted and documented, not hidden: - **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. The arithmetic assumes fleet-uniform `ThreadLockTTL` and `DetachTimeout` — the same uniform-fleet operator contract as capability participation: a short-timeout instance's registration could otherwise expire while a long-timeout instance's older fence is still within its horizon, reopening the empty-register reordering. Heterogeneous fleets must therefore derive every backend's registration TTL from fleet-wide maxima — for the per-call-TTL backends (Memory, Redis, Postgres) as much as for NATS's bucket TTL. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). - **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. -Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. +Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), where Memory and Postgres must purge expired register entries opportunistically or on a schedule — lazy same-key expiry alone would leak one register entry per one-off scope, the same unbounded-cardinality problem that rejected per-scope counters — and **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. ### 4. Preemption on a fenced Lock Takeover (`LockForcer` reshaped; issue #50's scope extension) From 2fb9b409f85f30aac18514e335dffac1fe8138f6 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 17:32:41 +0000 Subject: [PATCH 10/28] docs(adr): universal SHA-256 holder identity; reject burst + preemption hook --- docs/adr/0015-runtime-coordination.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 6c64fdf..b3b2e17 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -110,8 +110,9 @@ ADR 0012 proposed `ForceReleaseLock` by key. The #38 history shows why that shap ```go // LockHolder identifies a Thread Lock holder for compare-and-take. Identity is -// an opaque per-lease identifier, stable across ExtendLock refreshes; it need -// not be — and should not be — a release-capable token. +// universally the lowercase hex SHA-256 of the lease token: stable across +// ExtendLock refreshes, computable by any caller from a token it generated, +// and non-reversible — ObserveLock never exposes release capability. type LockHolder struct { Key string Identity string @@ -119,10 +120,10 @@ type LockHolder struct { // LockForcer is an optional State capability enabling preemption. type LockForcer interface { - // ObserveLock reports the current holder of key, if held. A holder - // installed via TakeLock must be recognizable to its caller through - // ObserveLock (identity equal to, or a specified derivation of, the - // caller-generated replacement token). + // ObserveLock reports the current holder of key, if held. Because Identity + // is the universal SHA-256 derivation, any caller can recognize a holder it + // installed via TakeLock by hashing its own replacement token — no + // backend-specific comparison is needed. ObserveLock(ctx context.Context, key string) (LockHolder, bool, error) // TakeLock atomically replaces the lock iff its current holder is still // observed, installing the caller-generated replacement lease. taken=false @@ -134,7 +135,7 @@ type LockForcer interface { - **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The reconciliation read is itself fallible, so it must be retried — reads are idempotent — on an independently bounded context until the replacement is confirmed present or definitively absent, or a bounded reconciliation budget expires; an outcome still inconclusive after the budget falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. - **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded, with a refresh that cannot complete within its bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound. - **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (`ThreadLockTTL/2`; ~60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). -- **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). +- **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). It also rejects `ConcurrencyBurst`: a batch holds one lock on behalf of many members while the hook receives a single `*Event`, and batch-level preemption semantics — which member represents the batch, whether one member's decision force-kills every member's turn — are deliberately not defined. - **Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted. - **Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here. From 33cb6c1da031357bbcf583f20724804c74230e99 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 17:57:45 +0000 Subject: [PATCH 11/28] docs(adr): uniform pre/post-mark degradation rule; concrete refresh RPC bound (one refresh interval) --- docs/adr/0015-runtime-coordination.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index b3b2e17..6488241 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -86,7 +86,7 @@ Fences come from **one global, non-resetting sequence**, not per-scope counters: Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. -- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed. Past either bound, a marked delivery proceeds per-instance (unfenced or degraded, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. +- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed. Past either bound, the delivery proceeds per-instance (scope degraded, observed) — marked or not: window misses, like every `Coalescer` failure, never fail the prelude (see the failure rules below), because an optional capability must not turn into user-visible webhook errors or lost direct invocations. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. - **One ordering source.** When the capability is present, the **Waiter Fence** *is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders deliveries only where fences are absent altogether (capability absent, sync dispatch). Degradation is therefore **scope-wide, not per-delivery**: a `Coalescer` failure degrades that delivery's scope on that instance to local admission-sequence ordering — local supersession and the dispatch-time check are suspended for the scope's currently-admitted deliveries until they drain — so fenced and unfenced deliveries are never ordered against each other by two different sources (mixed populations admit no consistent order). A degraded scope's earlier registrations still stand for other instances until they expire, which is the already-documented weaker-globally degradation. - **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. - **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. @@ -100,7 +100,7 @@ The guarantee split, stated honestly: Failure modes accepted and documented, not hidden: - **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. The arithmetic assumes fleet-uniform `ThreadLockTTL` and `DetachTimeout` — the same uniform-fleet operator contract as capability participation: a short-timeout instance's registration could otherwise expire while a long-timeout instance's older fence is still within its horizon, reopening the empty-register reordering. Heterogeneous fleets must therefore derive every backend's registration TTL from fleet-wide maxima — for the per-call-TTL backends (Memory, Redis, Postgres) as much as for NATS's bucket TTL. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). -- **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. +- **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude, but the degradation rule is uniform on both sides of it: any `Coalescer` failure during the prelude — allocation, registration, or a window miss, whether a backend error or the request context ending, before or after the mark — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), where Memory and Postgres must purge expired register entries opportunistically or on a schedule — lazy same-key expiry alone would leak one register entry per one-off scope, the same unbounded-cardinality problem that rejected per-scope counters — and **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. @@ -133,8 +133,8 @@ type LockForcer interface { ``` - **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The reconciliation read is itself fallible, so it must be retried — reads are idempotent — on an independently bounded context until the replacement is confirmed present or definitively absent, or a bounded reconciliation budget expires; an outcome still inconclusive after the budget falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. -- **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded, with a refresh that cannot complete within its bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound. -- **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (`ThreadLockTTL/2`; ~60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). +- **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded at **one refresh interval** (`ThreadLockTTL/2`; conformance-validated), with a refresh that cannot complete within that bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound — at most `ThreadLockTTL` (two minutes at defaults). +- **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (each `ThreadLockTTL/2`, so within `ThreadLockTTL` — two minutes at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). - **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). It also rejects `ConcurrencyBurst`: a batch holds one lock on behalf of many members while the hook receives a single `*Event`, and batch-level preemption semantics — which member represents the batch, whether one member's decision force-kills every member's turn — are deliberately not defined. - **Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted. - **Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here. From caf7abef3a0d75d7a52511baec0a8ae840b5a3b0 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 18:09:32 +0000 Subject: [PATCH 12/28] docs(adr): takeover + reconciliation execute in the detached tail under the DetachTimeout budget --- docs/adr/0015-runtime-coordination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 6488241..dada595 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -132,7 +132,7 @@ type LockForcer interface { } ``` -- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The reconciliation read is itself fallible, so it must be retried — reads are idempotent — on an independently bounded context until the replacement is confirmed present or definitively absent, or a bounded reconciliation budget expires; an outcome still inconclusive after the budget falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. +- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The entire observe → take → reconcile sequence runs in the detached tail, *after* acknowledgement — never in the synchronous prelude — following the same ack-prompt rule the deferred queue path already uses, so no takeover or reconciliation latency can collide with the platform's acknowledgement deadline. The reconciliation read is itself fallible, so it must be retried — reads are idempotent — on independently bounded calls, with the whole sequence budgeted by the **Detached Work Context** (`DetachTimeout`), until the replacement is confirmed present or definitively absent; an outcome still inconclusive when that budget ends falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. - **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded at **one refresh interval** (`ThreadLockTTL/2`; conformance-validated), with a refresh that cannot complete within that bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound — at most `ThreadLockTTL` (two minutes at defaults). - **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (each `ThreadLockTTL/2`, so within `ThreadLockTTL` — two minutes at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). - **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). It also rejects `ConcurrencyBurst`: a batch holds one lock on behalf of many members while the hook receives a single `*Event`, and batch-level preemption semantics — which member represents the batch, whether one member's decision force-kills every member's turn — are deliberately not defined. From 9fdf059f1dac45d9e4dbda9d5de2483c8765e30c Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 18:18:58 +0000 Subject: [PATCH 13/28] docs(adr): extend-as-reconciliation-probe with TTL horizon; ack-budget bounds on prelude fence calls --- docs/adr/0015-runtime-coordination.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index dada595..8f20657 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -86,7 +86,7 @@ Fences come from **one global, non-resetting sequence**, not per-scope counters: Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. -- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed. Past either bound, the delivery proceeds per-instance (scope degraded, observed) — marked or not: window misses, like every `Coalescer` failure, never fail the prelude (see the failure rules below), because an optional capability must not turn into user-visible webhook errors or lost direct invocations. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. +- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed, and they bound ordering *validity*, not latency: each prelude fence call (`NextFence`, `RegisterWaiter`) additionally carries a short per-call deadline derived from the acknowledgement budget (ADR 0002's fast-ack contract — a two-minute ordering window must never hold a three-second platform acknowledgement hostage), and a call that cannot complete within it degrades the scope per-instance *before* the acknowledgement deadline is at risk. Past either ordering bound, the delivery likewise proceeds per-instance (scope degraded, observed) — marked or not: window misses, like every `Coalescer` failure, never fail the prelude (see the failure rules below), because an optional capability must not turn into user-visible webhook errors or lost direct invocations. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. - **One ordering source.** When the capability is present, the **Waiter Fence** *is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders deliveries only where fences are absent altogether (capability absent, sync dispatch). Degradation is therefore **scope-wide, not per-delivery**: a `Coalescer` failure degrades that delivery's scope on that instance to local admission-sequence ordering — local supersession and the dispatch-time check are suspended for the scope's currently-admitted deliveries until they drain — so fenced and unfenced deliveries are never ordered against each other by two different sources (mixed populations admit no consistent order). A degraded scope's earlier registrations still stand for other instances until they expire, which is the already-documented weaker-globally degradation. - **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. - **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. @@ -132,7 +132,7 @@ type LockForcer interface { } ``` -- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The entire observe → take → reconcile sequence runs in the detached tail, *after* acknowledgement — never in the synchronous prelude — following the same ack-prompt rule the deferred queue path already uses, so no takeover or reconciliation latency can collide with the platform's acknowledgement deadline. The reconciliation read is itself fallible, so it must be retried — reads are idempotent — on independently bounded calls, with the whole sequence budgeted by the **Detached Work Context** (`DetachTimeout`), until the replacement is confirmed present or definitively absent; an outcome still inconclusive when that budget ends falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. +- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The entire observe → take → reconcile sequence runs in the detached tail, *after* acknowledgement — never in the synchronous prelude — following the same ack-prompt rule the deferred queue path already uses, so no takeover or reconciliation latency can collide with the platform's acknowledgement deadline. Reconciliation probes ownership by *extending* the replacement lease (`ExtendLock` with the caller's replacement): success both confirms the commit and refreshes the lease, so the replacement cannot expire mid-reconciliation. Probes are retried on independently bounded calls within the **Detached Work Context** budget (`DetachTimeout`); a failed probe is confirmed as non-commit only while the replacement's TTL horizon (monotonic, from the takeover attempt) has not elapsed — absence observed after that horizon can be a committed-then-expired takeover and is never treated as proof of non-commit. An outcome still inconclusive when the budget or horizon ends falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. - **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded at **one refresh interval** (`ThreadLockTTL/2`; conformance-validated), with a refresh that cannot complete within that bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound — at most `ThreadLockTTL` (two minutes at defaults). - **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (each `ThreadLockTTL/2`, so within `ThreadLockTTL` — two minutes at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). - **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). It also rejects `ConcurrencyBurst`: a batch holds one lock on behalf of many members while the hook receives a single `*Event`, and batch-level preemption semantics — which member represents the batch, whether one member's decision force-kills every member's turn — are deliberately not defined. From 08c6b028dcac8c7597c9b284415e4c060b273937 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 18:33:33 +0000 Subject: [PATCH 14/28] docs(adr): conflict-time observation binding, hook in tail, extend-only probe, read-completion validity, allocation before all prelude State reads, spec-level scoping --- docs/adr/0015-runtime-coordination.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 8f20657..55211a9 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -27,6 +27,8 @@ One domain constraint is honored throughout: **Runtime State** is coordination s Four pieces, decided together. Two are runtime-local (admission bound, burst shaping); two are optional **Runtime State** capabilities discovered by type assertion, per the ADR 0009 `HistoryReader` precedent — the required `State` interface does not change. +Specification level: this ADR pins interfaces, semantics, failure dispositions, and the named ordering invariants. The concrete bounds it states (window sizes, per-call deadlines, TTL formulas) are normative defaults chosen to satisfy those invariants; an implementation may refine a bound provided the invariant it serves still holds, and the conformance suite validates the invariants, not the constants. + ### 1. Admission Bound (issue #44) Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot at acknowledgement; only routed deferred work holds it to the tail's end. @@ -86,10 +88,10 @@ Fences come from **one global, non-resetting sequence**, not per-scope counters: Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. -- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed, and they bound ordering *validity*, not latency: each prelude fence call (`NextFence`, `RegisterWaiter`) additionally carries a short per-call deadline derived from the acknowledgement budget (ADR 0002's fast-ack contract — a two-minute ordering window must never hold a three-second platform acknowledgement hostage), and a call that cannot complete within it degrades the scope per-instance *before* the acknowledgement deadline is at risk. Past either ordering bound, the delivery likewise proceeds per-instance (scope degraded, observed) — marked or not: window misses, like every `Coalescer` failure, never fail the prelude (see the failure rules below), because an optional capability must not turn into user-visible webhook errors or lost direct invocations. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. +- **Allocate early; register within a bounded window.** Under queue/debounce, a delivery allocates its fence before *any other* prelude State round-trip — subscription-routing reads, the dedupe mark, and lock acquisition are all reorderable stalls (invariant: no State call an older delivery can stall on may precede fence allocation). Deliveries that turn out unrouted or duplicate waste their fence harmlessly, since allocation alone supersedes no one. This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed, and they bound ordering *validity*, not latency: each prelude fence call (`NextFence`, `RegisterWaiter`) additionally carries a short per-call deadline derived from the acknowledgement budget (ADR 0002's fast-ack contract — a two-minute ordering window must never hold a three-second platform acknowledgement hostage), and a call that cannot complete within it degrades the scope per-instance *before* the acknowledgement deadline is at risk. Past either ordering bound, the delivery likewise proceeds per-instance (scope degraded, observed) — marked or not: window misses, like every `Coalescer` failure, never fail the prelude (see the failure rules below), because an optional capability must not turn into user-visible webhook errors or lost direct invocations. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. - **One ordering source.** When the capability is present, the **Waiter Fence** *is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders deliveries only where fences are absent altogether (capability absent, sync dispatch). Degradation is therefore **scope-wide, not per-delivery**: a `Coalescer` failure degrades that delivery's scope on that instance to local admission-sequence ordering — local supersession and the dispatch-time check are suspended for the scope's currently-admitted deliveries until they drain — so fenced and unfenced deliveries are never ordered against each other by two different sources (mixed populations admit no consistent order). A degraded scope's earlier registrations still stand for other instances until they expire, which is the already-documented weaker-globally degradation. - **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. -- **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. +- **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. The read carries its own per-call bound, and its result is trusted only if it *completes* within the check horizon — a read that fails, or returns past the horizon, degrades the scope (observed) rather than being taken as evidence of absence, since a newer registration could have expired while the read was in flight. If a newer fence is registered, the delivery skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. The guarantee split, stated honestly: @@ -120,10 +122,9 @@ type LockHolder struct { // LockForcer is an optional State capability enabling preemption. type LockForcer interface { - // ObserveLock reports the current holder of key, if held. Because Identity - // is the universal SHA-256 derivation, any caller can recognize a holder it - // installed via TakeLock by hashing its own replacement token — no - // backend-specific comparison is needed. + // ObserveLock reports the current holder of key, if held. Identity is the + // universal SHA-256 derivation, so any caller can compare holders without + // backend-specific logic. ObserveLock(ctx context.Context, key string) (LockHolder, bool, error) // TakeLock atomically replaces the lock iff its current holder is still // observed, installing the caller-generated replacement lease. taken=false @@ -132,10 +133,10 @@ type LockForcer interface { } ``` -- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The entire observe → take → reconcile sequence runs in the detached tail, *after* acknowledgement — never in the synchronous prelude — following the same ack-prompt rule the deferred queue path already uses, so no takeover or reconciliation latency can collide with the platform's acknowledgement deadline. Reconciliation probes ownership by *extending* the replacement lease (`ExtendLock` with the caller's replacement): success both confirms the commit and refreshes the lease, so the replacement cannot expire mid-reconciliation. Probes are retried on independently bounded calls within the **Detached Work Context** budget (`DetachTimeout`); a failed probe is confirmed as non-commit only while the replacement's TTL horizon (monotonic, from the takeover attempt) has not elapsed — absence observed after that horizon can be a committed-then-expired takeover and is never treated as proof of non-commit. An outcome still inconclusive when the budget or horizon ends falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched. +- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconcilable instead of leaving an orphaned lock stalling the scope until TTL. The sequencing binds the takeover to the conflict, not to whoever holds the lock later: `ObserveLock` runs *at the Lock Conflict itself*, in the prelude, under the same ack-budget per-call bound as the fence calls — capturing the holder identity observed at conflict time, exactly the compare-and-invalidate binding #50 requires — while the takeover and its reconciliation run in the detached tail, after acknowledgement, so their latency can never collide with the platform's acknowledgement deadline. Because the tail's `TakeLock` can only present the conflict-time identity, a holder that changed in between (the victim finished; a successor acquired) fails the compare and falls back to the strategy path: a preemptor can never take a successor's lease. Reconciliation of an *ambiguous* `TakeLock` probes ownership by **extending** the replacement lease (`ExtendLock` with the caller's replacement) — never by observation, which cannot renew the replacement and would let it expire mid-investigation: a successful extend both confirms the commit and refreshes the lease. Probes are retried on independently bounded calls within the **Detached Work Context** budget (`DetachTimeout`); a failed probe is confirmed as non-commit only while the replacement's TTL horizon (monotonic, from the takeover attempt) has not elapsed — absence observed after that horizon can be a committed-then-expired takeover and is never treated as proof of non-commit. An outcome still inconclusive when the budget or horizon ends falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Failure dispositions are position-dependent, mirroring §3: a prelude `ObserveLock` failure (backend error or ack-budget miss) degrades to the strategy path without failing the marked prelude; in the tail, a backend failure while the tail context is live falls back observably to the strategy path, while cancellation or expiry of the tail's own context is the existing abandonment path — `DetachTimeout` has exhausted the delivery's budget, and fallback must not launch a handler past it. - **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded at **one refresh interval** (`ThreadLockTTL/2`; conformance-validated), with a refresh that cannot complete within that bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound — at most `ThreadLockTTL` (two minutes at defaults). - **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (each `ThreadLockTTL/2`, so within `ThreadLockTTL` — two minutes at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). -- **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). It also rejects `ConcurrencyBurst`: a batch holds one lock on behalf of many members while the hook receives a single `*Event`, and batch-level preemption semantics — which member represents the batch, whether one member's decision force-kills every member's turn — are deliberately not defined. +- **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted for a delivery that hit a **Lock Conflict** — but it executes in the detached tail, after acknowledgement, never in the synchronous prelude: the hook is application code that may compute or perform I/O, and it must not be able to hold a platform acknowledgement hostage. True → fenced takeover of the conflict-time-observed holder; false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). It also rejects `ConcurrencyBurst`: a batch holds one lock on behalf of many members while the hook receives a single `*Event`, and batch-level preemption semantics — which member represents the batch, whether one member's decision force-kills every member's turn — are deliberately not defined. - **Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted. - **Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here. From 47d34bc846bdfd8180cf7afbd16d90df8814cf04 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 18:46:09 +0000 Subject: [PATCH 15/28] docs(adr): descope preemption to rejected shapes + binding requirements + deferred protocol design (non-converging surface, per the #38 precedent) --- docs/adr/0015-runtime-coordination.md | 47 +++++++++++++++------------ 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 55211a9..66ea362 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -91,7 +91,7 @@ Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is - **Allocate early; register within a bounded window.** Under queue/debounce, a delivery allocates its fence before *any other* prelude State round-trip — subscription-routing reads, the dedupe mark, and lock acquisition are all reorderable stalls (invariant: no State call an older delivery can stall on may precede fence allocation). Deliveries that turn out unrouted or duplicate waste their fence harmlessly, since allocation alone supersedes no one. This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed, and they bound ordering *validity*, not latency: each prelude fence call (`NextFence`, `RegisterWaiter`) additionally carries a short per-call deadline derived from the acknowledgement budget (ADR 0002's fast-ack contract — a two-minute ordering window must never hold a three-second platform acknowledgement hostage), and a call that cannot complete within it degrades the scope per-instance *before* the acknowledgement deadline is at risk. Past either ordering bound, the delivery likewise proceeds per-instance (scope degraded, observed) — marked or not: window misses, like every `Coalescer` failure, never fail the prelude (see the failure rules below), because an optional capability must not turn into user-visible webhook errors or lost direct invocations. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. - **One ordering source.** When the capability is present, the **Waiter Fence** *is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders deliveries only where fences are absent altogether (capability absent, sync dispatch). Degradation is therefore **scope-wide, not per-delivery**: a `Coalescer` failure degrades that delivery's scope on that instance to local admission-sequence ordering — local supersession and the dispatch-time check are suspended for the scope's currently-admitted deliveries until they drain — so fenced and unfenced deliveries are never ordered against each other by two different sources (mixed populations admit no consistent order). A degraded scope's earlier registrations still stand for other instances until they expire, which is the already-documented weaker-globally degradation. - **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. -- **Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. The read carries its own per-call bound, and its result is trusted only if it *completes* within the check horizon — a read that fails, or returns past the horizon, degrades the scope (observed) rather than being taken as evidence of absence, since a newer registration could have expired while the read was in flight. If a newer fence is registered, the delivery skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. +- **Check on dispatch — the turn-claim point.** Every registered delivery re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. The read carries its own per-call bound, and its result is trusted only if it *completes* within the check horizon — a read that fails, or returns past the horizon, degrades the scope (observed) rather than being taken as evidence of absence, since a newer registration could have expired while the read was in flight. If a newer fence is registered, the delivery skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. The guarantee split, stated honestly: @@ -106,39 +106,44 @@ Failure modes accepted and documented, not hidden: Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), where Memory and Postgres must purge expired register entries opportunistically or on a schedule — lazy same-key expiry alone would leak one register entry per one-off scope, the same unbounded-cardinality problem that rejected per-scope counters — and **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. -### 4. Preemption on a fenced Lock Takeover (`LockForcer` reshaped; issue #50's scope extension) +### 4. Preemption (issue #50's scope extension): rejected shapes, binding requirements, deferred protocol -ADR 0012 proposed `ForceReleaseLock` by key. The #38 history shows why that shape cannot be made safe (classes 3–5): key-only identity cannot distinguish victim from successor, release-then-acquire leaves a gap a third party can enter, and the compensating local choreography (`inflightCancels`, `victimDone`) was the single largest P1 source. The reshaped surface: +ADR 0012 proposed `ForceReleaseLock` by key. The #38 history shows why that shape cannot be made safe (classes 3–5): key-only identity cannot distinguish victim from successor, release-then-acquire leaves a gap a third party can enter, and the compensating local choreography (`inflightCancels`, `victimDone`) was the single largest P1 source. **Rejected outright, finally:** key-only `ForceReleaseLock(ctx, key)` and its four staged backend implementations; the `inflightCancels` registry; `preemptLocalIfPending` and the `victimDone` victim-drain handshake. + +What replaces it is decided here at the contract level — a fenced **Lock Takeover** — while the full dispatch-integration protocol is deliberately deferred to a dedicated follow-up design. This ADR's own review demonstrated why: the takeover protocol couples to every dispatch phase and strategy (prelude vs tail placement, acknowledgement budgets, per-strategy conflict-discovery points) and to ADR 0002's accepted rule that the **Thread Lock** is acquired in the prelude, never after acknowledgement. Pinning that protocol piecemeal here reproduces the churn that parked PR #53; it must be designed once, against the dispatch structure as a whole. + +The capability sketch consistent with the requirements below (illustrative; the follow-up design finalizes the shape — in particular, the conflict-time observation most likely folds into the acquisition primitive as acquire-or-observe rather than a separate read): ```go // LockHolder identifies a Thread Lock holder for compare-and-take. Identity is // universally the lowercase hex SHA-256 of the lease token: stable across // ExtendLock refreshes, computable by any caller from a token it generated, -// and non-reversible — ObserveLock never exposes release capability. +// and non-reversible — observation never exposes release capability. type LockHolder struct { Key string Identity string } // LockForcer is an optional State capability enabling preemption. +// TakeLock atomically replaces the lock iff its current holder is still the +// observed one, installing the caller-generated replacement lease. +// taken=false means the holder changed or the lock is free — never an error. type LockForcer interface { - // ObserveLock reports the current holder of key, if held. Identity is the - // universal SHA-256 derivation, so any caller can compare holders without - // backend-specific logic. - ObserveLock(ctx context.Context, key string) (LockHolder, bool, error) - // TakeLock atomically replaces the lock iff its current holder is still - // observed, installing the caller-generated replacement lease. taken=false - // means the holder changed or the lock is free — never an error. TakeLock(ctx context.Context, observed LockHolder, replacement LockLease, ttl time.Duration) (bool, error) } ``` -- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconcilable instead of leaving an orphaned lock stalling the scope until TTL. The sequencing binds the takeover to the conflict, not to whoever holds the lock later: `ObserveLock` runs *at the Lock Conflict itself*, in the prelude, under the same ack-budget per-call bound as the fence calls — capturing the holder identity observed at conflict time, exactly the compare-and-invalidate binding #50 requires — while the takeover and its reconciliation run in the detached tail, after acknowledgement, so their latency can never collide with the platform's acknowledgement deadline. Because the tail's `TakeLock` can only present the conflict-time identity, a holder that changed in between (the victim finished; a successor acquired) fails the compare and falls back to the strategy path: a preemptor can never take a successor's lease. Reconciliation of an *ambiguous* `TakeLock` probes ownership by **extending** the replacement lease (`ExtendLock` with the caller's replacement) — never by observation, which cannot renew the replacement and would let it expire mid-investigation: a successful extend both confirms the commit and refreshes the lease. Probes are retried on independently bounded calls within the **Detached Work Context** budget (`DetachTimeout`); a failed probe is confirmed as non-commit only while the replacement's TTL horizon (monotonic, from the takeover attempt) has not elapsed — absence observed after that horizon can be a committed-then-expired takeover and is never treated as proof of non-commit. An outcome still inconclusive when the budget or horizon ends falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Failure dispositions are position-dependent, mirroring §3: a prelude `ObserveLock` failure (backend error or ack-budget miss) degrades to the strategy path without failing the marked prelude; in the tail, a backend failure while the tail context is live falls back observably to the strategy path, while cancellation or expiry of the tail's own context is the existing abandonment path — `DetachTimeout` has exhausted the delivery's budget, and fallback must not launch a handler past it. -- **No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded at **one refresh interval** (`ThreadLockTTL/2`; conformance-validated), with a refresh that cannot complete within that bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound — at most `ThreadLockTTL` (two minutes at defaults). -- **What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (each `ThreadLockTTL/2`, so within `ThreadLockTTL` — two minutes at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on). -- **Hook and gating.** `RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted for a delivery that hit a **Lock Conflict** — but it executes in the detached tail, after acknowledgement, never in the synchronous prelude: the hook is application code that may compute or perform I/O, and it must not be able to hold a platform acknowledgement hostage. True → fenced takeover of the conflict-time-observed holder; false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy). It also rejects `ConcurrencyBurst`: a batch holds one lock on behalf of many members while the hook receives a single `*Event`, and batch-level preemption semantics — which member represents the batch, whether one member's decision force-kills every member's turn — are deliberately not defined. -- **Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted. -- **Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here. +**Binding requirements** — this ADR's contract for the follow-up design; nothing weaker ships: + +1. **Takeover, not release-then-acquire.** A single compare-and-swap on the lock key: no window in which a third party can acquire between "release" and "acquire" (class 3 precluded by construction). A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must, and then the preemptor falls back to the configured strategy path, observed — no retry loop. +2. **Conflict-time binding, captured atomically.** The takeover may only present the holder identity captured *atomically with the failed acquisition that constituted the Lock Conflict* — an acquire-or-observe primitive, not a separate later read, which reintroduces the successor race (a victim finishing and a successor acquiring between conflict and observation). A preemptor must be structurally unable to take a successor's lease. +3. **Idempotent, renewing reconciliation.** The replacement lease token is caller-generated. An ambiguous takeover (committed, response lost) is reconciled by *extending* the replacement — confirming and renewing in one probe — never by observation alone, which cannot renew and lets the replacement expire mid-investigation; absence observed after the replacement's TTL horizon is never proof of non-commit. Inconclusive outcomes fall back with their own observation as a documented backend-outage residual. +4. **One path, no local registries.** Preemption goes through the State whether the victim is local or remote (classes 4 and 5 precluded: no process-map/State agreement problem, no victim-drain handshake). The victim stops via the universal lease-loss cancellation already on main, with every lease-refresh call independently time-bounded at one refresh interval (a refresh that cannot complete counts as lease loss — today's `ExtendLock` under `context.WithoutCancel` has no RPC deadline). Signal latency: at most one refresh interval plus the per-call bound (`ThreadLockTTL` total at defaults); termination is cooperative — Go cannot forcibly stop a handler — and a handler that ignores cancellation overlaps unboundedly, the same contract as `DetachTimeout`, shutdown drain, and lease loss. A local-victim nudge may only ever be best-effort latency sugar, never correctness machinery. +5. **The acknowledgement is never hostage.** Neither the `OnLockConflict` hook (application code) nor takeover/reconciliation latency may gate the platform acknowledgement. +6. **Fail-fast gating.** Preemption requires `DispatchDeferred` (a sync holder cannot be stopped), a State implementing `LockForcer` (constructor error otherwise — a preemption hook on a State that cannot preempt is a misconfiguration, not a degradation), and a strategy whose conflict point is well-defined for the hook: `ConcurrencyConcurrent` (no lock, no conflict) and `ConcurrencyBurst` (one lock for many members, one `*Event` in the hook) are rejected at construction. Uniform fleet configuration is an operator contract, as in §3. +7. **ADR 0002 reconciled explicitly.** ADR 0002 requires the Thread Lock acquired in the prelude and rejects post-acknowledgement acquisition. Any protocol that swaps or acquires the lock after acknowledgement reopens that rule and must supersede it explicitly in the follow-up design — never silently. + +**Open questions the follow-up design must resolve before any preemption implementation:** the acquire-or-observe primitive's exact shape on the required-vs-optional interface boundary; the takeover's placement relative to ADR 0002's prelude-lock rule; the conflict-observation point for strategies that discover contention in the tail (debounce takes no prelude lock — define its observation point or reject debounce + preemption); and the interaction between takeover and the §3 fence check for conflicted preemptors. Until that design is accepted, the force/steerability surface stays reserved, exactly as on main today. ### 5. Composition @@ -156,14 +161,14 @@ type LockForcer interface { | 6 | Lease-lifecycle divergence | Precluded by rule: all lock-holding paths (including burst) reuse the one refresh/cancel/outcome implementation | | 7 | Temporal/budget skew | Precluded: early fence allocation pins cross-instance admission order; per-member execution budgets; no shared batch deadline | -Accepted residuals, all observable: coalesced-turn loss on register-then-die (§3); preemption overlap of one refresh interval plus the victim's cooperative cancellation latency (§4); bounded spurious preemption by a superseded preemptor (§4); fence order approximating State-arrival order, not platform send order (§3); per-instance degradation for deliveries whose prelude stalls past the registration window (§3). +Accepted residuals, all observable: coalesced-turn loss on register-then-die (§3); preemption overlap of the cancellation-signal latency plus the victim's cooperative cancellation latency (§4, requirement 4); fence order approximating State-arrival order, not platform send order (§3); per-instance degradation for deliveries whose prelude stalls past the registration window (§3). Preemption's remaining residuals are finalized by §4's follow-up design within its binding requirements. ## Outcome for PR #53 (staged burst + preemption) **Verdict: close PR #53.** Neither half should merge in its current shape, and the halves should not travel together again. - **Burst: rewrite on this design, salvaging the concept and its tests.** `ConcurrencyBurst` and per-member fresh budgets survive. Required changes: members count against `MaxDetached`; `MaxBurstBatch` with seal-and-roll; the batch lock lifecycle must reuse the shared refresh/cancel machinery (the staged branch's separate refresh loop reproduced class 6 twice). Ship as its own PR after the admission bound lands. -- **Preemption: rewrite from scratch.** Rejected outright: key-only `ForceReleaseLock(ctx, key)` and all four backend implementations of it; the `inflightCancels` registry; `preemptLocalIfPending`/`victimDone`. Surviving with changed shape: `OnLockConflict` (same signature, now deferred-only and `LockForcer`-required at construction), `ErrPreempted`/`OutcomePreempted` (already on main), and the `LockForcer` capability name — carrying the `ObserveLock`/`TakeLock` contract above. Ship last, after the `Coalescer` fence exists, so the stale-preemptor check has something to read. +- **Preemption: close outright; revival is gated on §4's follow-up design.** Rejected finally: key-only `ForceReleaseLock(ctx, key)` and all four backend implementations of it; the `inflightCancels` registry; `preemptLocalIfPending`/`victimDone`. Surviving as reserved names under §4's binding requirements: `OnLockConflict` (deferred-only, `LockForcer`-required, concurrent/burst rejected), `ErrPreempted`/`OutcomePreempted` (already on main), and the `LockForcer` fenced-takeover capability. No preemption PR is acceptable until the dedicated protocol design resolves §4's open questions; it ships last, after the `Coalescer` fence exists. ## Non-goals @@ -183,10 +188,10 @@ This design explicitly refuses to promise: - Two new **Runtime Options** (`MaxDetached`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. - Adapters gain one honesty duty: map `ErrAdmissionRejected` shape-awarely — a retry-inducing response for platform-redelivered shapes, a truthful busy response for direct invocations the platform does not redeliver. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. -- The `State` interface itself does not change; both extensions are optional capabilities (ADR 0009 precedent). All four in-repo States implement both; the conformance suite grows capability sections (fence monotonicity, register-only-newer, TTL expiry, observe/take atomicity under contention, extend-churn non-defeat). +- The `State` interface itself does not change; both extensions are optional capabilities (ADR 0009 precedent). All four in-repo States implement `Coalescer` when its integration ships; `LockForcer` implementations follow §4's follow-up design. The conformance suite grows capability sections (fence monotonicity, register-only-newer, TTL expiry; for takeover later: compare-and-swap atomicity under contention, extend-churn non-defeat). - Queue/debounce dispatch under `DispatchDeferred` gains up to three State round-trips (allocate, register, check) when `Coalescer` is present — the cost of global newest-wins. Absent the capability, or under sync dispatch, nothing new is paid. - ADR 0012's proposed force surface is superseded by §4 (flagged per the domain docs' ADR-conflict rule). ADR 0012's status note and the CONTEXT.md glossary (**Admission Bound**, **Waiter Fence**, **Lock Takeover**) update when this ADR is accepted and implementation lands. -- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) `Coalescer` + backends + conformance; (3) queue/debounce fence integration — closes #50; (4) burst revival; (5) fenced preemption, including the independently time-bounded lease-refresh calls. +- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) `Coalescer` + backends + conformance; (3) queue/debounce fence integration — closes #50; (4) burst revival; (5) the dedicated preemption-protocol design resolving §4's open questions within its binding requirements, then the fenced-preemption implementation (including the independently time-bounded lease-refresh calls). ## Alternatives Considered From 20d00562d8d8b9a08cc5f3ac31f7ebcbd2f801af Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 18:57:43 +0000 Subject: [PATCH 16/28] =?UTF-8?q?docs(adr):=20re-scope=20per=20maintainer?= =?UTF-8?q?=20directive=20=E2=80=94=20admission=20bound=20kept,=20cross-in?= =?UTF-8?q?stance=20coalescing=20rejected=20for=20now=20with=20formal-desi?= =?UTF-8?q?gn=20reopening=20bar?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/adr/0015-runtime-coordination.md | 192 ++++++-------------------- 1 file changed, 45 insertions(+), 147 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 66ea362..f717db9 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -1,34 +1,22 @@ -# ADR 0015: Runtime Coordination — Admission Bound and Fenced Cross-Instance Coalescing +# ADR 0015: Deferred-Dispatch Admission Bound; Cross-Instance Coalescing Rejected For Now ## Status -Proposed. This is the design for the deferred-dispatch admission bound (issue #44) and cross-instance coalescing (issue #50), decided together as one runtime-coordination track. It reshapes the force/steerability surface ADR 0012 proposed (`ForceReleaseLock` by key) and gives the staged burst/preemption branch (PR #53) its verdict. +Proposed. This decides the deferred-dispatch admission bound (issue #44) and explicitly rejects, for now, the cross-instance coalescing State extension (issue #50) after a full design attempt — see the rejection section for the evidence and the reopening bar. Issue #50 stays open as gated future work. This ADR also gives the staged burst/preemption branch (PR #53) its verdict. ## Context -ADR 0002 made `DispatchDeferred` the opt-in **Ack-Then-Work** mode: the adapter acknowledges after the synchronous prelude and the handler runs on the **Detached Work Context**, bounded by `DetachTimeout`. ADR 0012 expanded the **Concurrency Strategy** set; the reduced implementation on main ships `drop`, `queue`, `debounce`, and `concurrent` plus **Lock Scope** and universal lease-loss cancellation (a deferred holder whose **Lock Lease** is lost mid-run is cancelled within one refresh interval, `ThreadLockTTL/2`). Two costs were deliberately deferred: +ADR 0002 made `DispatchDeferred` the opt-in **Ack-Then-Work** mode: the adapter acknowledges after the synchronous prelude and the handler runs on the **Detached Work Context**, bounded by `DetachTimeout`. ADR 0012 expanded the **Concurrency Strategy** set; the reduced implementation on main ships `drop`, `queue`, `debounce`, and `concurrent` plus **Lock Scope** and universal lease-loss cancellation, with `burst` and force/steerability staged on PR #53 pending this design track. -- **No admission bound (#44).** Every accepted routed event under `DispatchDeferred` retains a detached tail — goroutine, event payload, handler closure — until completion, supersession, or `DetachTimeout`. A unique-event flood grows retention linearly. Burst batch members would retain payloads without even holding a goroutine. -- **Per-instance coalescing (#50).** Queue supersession and debounce coalescing live in process memory (the pending-waiter registry). Instances sharing a production **Runtime State** each dispatch their own most-recent event; the **Thread Lock** serializes them, but nothing supersedes across instances, while the upstream semantic is global newest-wins. +Two costs were deliberately deferred from that reduction: -The burst strategy and the `OnLockConflict`/`ForceReleaseLock` preemption path were staged on PR #53 rather than merged, because nine review rounds on PR #38 kept finding P1s in exactly that machinery. Those findings were not random; they reduce to recurring failure classes, and this design's job is to preclude the classes structurally rather than patch instances of them: +- **No admission bound (#44).** Every accepted routed event under `DispatchDeferred` retains a detached tail — goroutine, event payload, handler closure — until completion, supersession, or `DetachTimeout`. A unique-event flood grows retention linearly ([r3871060076](https://github.com/coder/chat/pull/38#discussion_r3871060076)). Burst batch members would retain payloads without even holding a goroutine ([r3871214506](https://github.com/coder/chat/pull/38#discussion_r3871214506)). +- **Per-instance coalescing (#50).** Queue supersession and debounce coalescing live in process memory: instances sharing a production **Runtime State** each dispatch their own most-recent event, serialized by the **Thread Lock** but not superseded across instances ([r3871403308](https://github.com/coder/chat/pull/38#discussion_r3871403308)). -1. **Unbounded admission** — goroutines/payloads retained before any concurrency gate ([r3871060076](https://github.com/coder/chat/pull/38#discussion_r3871060076), [r3871214506](https://github.com/coder/chat/pull/38#discussion_r3871214506)). -2. **Process-local coordination with distributed semantics implied** — per-instance registries (waiters, cancels) that multi-instance deployments silently defeat ([r3871403308](https://github.com/coder/chat/pull/38#discussion_r3871403308), [r3871919968](https://github.com/coder/chat/pull/38#discussion_r3871919968)). -3. **Key-only force release** — deleting a lock by key alone races holder turnover and kills innocent successor leases ([r3871938987](https://github.com/coder/chat/pull/38#discussion_r3871938987)). -4. **Non-atomic ownership handoff** — choreography between lock acquisition, local registration maps, and supersession rechecks, each gap a time-of-check race ([r3872104899](https://github.com/coder/chat/pull/38#discussion_r3872104899), [r3872390326](https://github.com/coder/chat/pull/38#discussion_r3872390326)). -5. **Premature handover** — a preemptor proceeding while its victim still runs ([r3871040671](https://github.com/coder/chat/pull/38#discussion_r3871040671)). -6. **Lease-lifecycle divergence** — parallel refresh/cleanup loops re-deriving hardening the shared path already has ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204), [r3872390336](https://github.com/coder/chat/pull/38#discussion_r3872390336)). -7. **Temporal and budget skew** — shared batch deadlines starving late members; prelude stalls displacing newer waiters ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320), [r3872622515](https://github.com/coder/chat/pull/38#discussion_r3872622515)). - -One domain constraint is honored throughout: **Runtime State** is coordination state, not **Thread Application State** and not a message store (CONTEXT.md; the ADR 0009 **History Reader** storage rule). No design below places event payloads in State. +Both were drafted here as one design. The admission bound converged under review; the cross-instance protocol did not — this ADR records the resulting split decision. One domain constraint holds throughout: **Runtime State** is coordination state, not **Thread Application State** and not a message store (CONTEXT.md; the ADR 0009 **History Reader** storage rule). ## Decision -Four pieces, decided together. Two are runtime-local (admission bound, burst shaping); two are optional **Runtime State** capabilities discovered by type assertion, per the ADR 0009 `HistoryReader` precedent — the required `State` interface does not change. - -Specification level: this ADR pins interfaces, semantics, failure dispositions, and the named ordering invariants. The concrete bounds it states (window sizes, per-call deadlines, TTL formulas) are normative defaults chosen to satisfy those invariants; an implementation may refine a bound provided the invariant it serves still holds, and the conformance suite validates the invariants, not the constants. - ### 1. Admission Bound (issue #44) Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot at acknowledgement; only routed deferred work holds it to the tail's end. @@ -56,142 +44,72 @@ Interaction with each strategy (under `DispatchDeferred`): ### 2. Burst shaping (issue #44's scope extension) -Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below — members counted by `MaxDetached`, fresh per-member budgets — are defined only for deferred dispatch. When burst ships (see the PR #53 outcome), batch growth is bounded twice: +Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission to its member-run's terminal outcome. - **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) -Two rules carried from the failure classes: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member (class 7) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder (class 6): one lease-lifecycle implementation, no parallel loop. Byte-based accounting is refused — see Non-goals. - -### 3. Cross-instance coalescing (issue #50): Waiter Fences on an optional `Coalescer` capability - -Queue supersession and debounce coalescing extend across instances through a State-issued monotonic **Waiter Fence** — a fencing token that totally orders coalescing participants. New optional capability: - -```go -// Coalescer is an optional State capability providing fenced cross-instance -// waiter supersession. -type Coalescer interface { - // NextFence allocates the next fence from one non-resetting monotonic - // sequence shared by all scopes. Allocator state is O(1) and permanent; - // fences are never reissued. - NextFence(ctx context.Context) (uint64, error) - // RegisterWaiter records fence as the scope's newest waiter iff it is newer - // than the currently registered fence; registered=false means the caller is - // already superseded. - RegisterWaiter(ctx context.Context, scope string, fence uint64, ttl time.Duration) (bool, error) - // NewestWaiter reads the scope's currently registered fence, if any. - NewestWaiter(ctx context.Context, scope string) (uint64, bool, error) -} -``` - -Fences come from **one global, non-resetting sequence**, not per-scope counters: a per-scope counter would need a TTL to avoid unbounded per-scope State growth, and an expired-then-reset counter would let an old delivery's high fence outrank a fresh low one, reversing newest-wins. A global sequence keeps allocator state O(1) and permanent while per-scope *register* entries expire freely — an expired register only ever degrades to per-instance semantics, never reorders. - -Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`. - -- **Allocate early; register within a bounded window.** Under queue/debounce, a delivery allocates its fence before *any other* prelude State round-trip — subscription-routing reads, the dedupe mark, and lock acquisition are all reorderable stalls (invariant: no State call an older delivery can stall on may precede fence allocation). Deliveries that turn out unrouted or duplicate waste their fence harmlessly, since allocation alone supersedes no one. This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window — one `ThreadLockTTL`, enforced on a local monotonic clock whose deadline is anchored *before* the `NextFence` call, so RPC response latency counts against the window (the fence may commit at the State long before its response arrives, and a response received past the deadline is treated as degraded). The runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed, and they bound ordering *validity*, not latency: each prelude fence call (`NextFence`, `RegisterWaiter`) additionally carries a short per-call deadline derived from the acknowledgement budget (ADR 0002's fast-ack contract — a two-minute ordering window must never hold a three-second platform acknowledgement hostage), and a call that cannot complete within it degrades the scope per-instance *before* the acknowledgement deadline is at risk. Past either ordering bound, the delivery likewise proceeds per-instance (scope degraded, observed) — marked or not: window misses, like every `Coalescer` failure, never fail the prelude (see the failure rules below), because an optional capability must not turn into user-visible webhook errors or lost direct invocations. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran. -- **One ordering source.** When the capability is present, the **Waiter Fence** *is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders deliveries only where fences are absent altogether (capability absent, sync dispatch). Degradation is therefore **scope-wide, not per-delivery**: a `Coalescer` failure degrades that delivery's scope on that instance to local admission-sequence ordering — local supersession and the dispatch-time check are suspended for the scope's currently-admitted deliveries until they drain — so fenced and unfenced deliveries are never ordered against each other by two different sources (mixed populations admit no consistent order). A degraded scope's earlier registrations still stand for other instances until they expire, which is the already-documented weaker-globally degradation. -- **Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession. -- **Check on dispatch — the turn-claim point.** Every registered delivery re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. The read carries its own per-call bound, and its result is trusted only if it *completes* within the check horizon — a read that fails, or returns past the horizon, degrades the scope (observed) rather than being taken as evidence of absence, since a newer registration could have expired while the read was in flight. If a newer fence is registered, the delivery skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics. - -The guarantee split, stated honestly: - -- **Per-instance (unchanged, always):** most-recent-wins by admission order; superseded waiters exit promptly; skips are observable. -- **Cross-instance (capability present, deferred dispatch, uniform fleet):** among registered deliveries that have not yet claimed their turn, at most the newest dispatches. "Newest" is defined by fence-allocation order at the shared State, which approximates arrival order — not platform send order — under delivery skew. The guarantee assumes every instance sharing the State participates: during a rolling upgrade, deliveries handled by non-participating instances (older runtimes, or runtimes whose State lacks the capability) dispatch unfenced and are invisible to fenced ordering — cross-instance newest-wins is suspended for exactly those deliveries and resumes once the fleet is uniform; fenced deliveries still order among themselves. A capability/version gate that could enforce fleet uniformity from within the runtime is refused (it would need fleet membership in State); uniformity is an operator rollout contract, documented, like the preemption mixed-fleet rule in §4. -- **Capability absent (or sync dispatch):** exactly today's documented per-instance behavior — correct per instance, weaker globally, logged once at startup. Never a constructor error: per-instance coalescing is valid semantics, not a broken configuration. - -Failure modes accepted and documented, not hidden: - -- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. The arithmetic assumes fleet-uniform `ThreadLockTTL` and `DetachTimeout` — the same uniform-fleet operator contract as capability participation: a short-timeout instance's registration could otherwise expire while a long-timeout instance's older fence is still within its horizon, reopening the empty-register reordering. Heterogeneous fleets must therefore derive every backend's registration TTL from fleet-wide maxima — for the per-call-TTL backends (Memory, Redis, Postgres) as much as for NATS's bucket TTL. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work). -- **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude, but the degradation rule is uniform on both sides of it: any `Coalescer` failure during the prelude — allocation, registration, or a window miss, whether a backend error or the request context ending, before or after the mark — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today. - -Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), where Memory and Postgres must purge expired register entries opportunistically or on a schedule — lazy same-key expiry alone would leak one register entry per one-off scope, the same unbounded-cardinality problem that rejected per-scope counters — and **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics. +Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Byte-based accounting is refused — see Non-goals. -### 4. Preemption (issue #50's scope extension): rejected shapes, binding requirements, deferred protocol +### 3. Per-instance supersession is the v0.x contract -ADR 0012 proposed `ForceReleaseLock` by key. The #38 history shows why that shape cannot be made safe (classes 3–5): key-only identity cannot distinguish victim from successor, release-then-acquire leaves a gap a third party can enter, and the compensating local choreography (`inflightCancels`, `victimDone`) was the single largest P1 source. **Rejected outright, finally:** key-only `ForceReleaseLock(ctx, key)` and its four staged backend implementations; the `inflightCancels` registry; `preemptLocalIfPending` and the `victimDone` victim-drain handshake. +Queue supersession and debounce coalescing are per runtime instance, by decision and not merely by implementation status: most-recent-wins follows the local dispatch admission sequence, superseded waiters exit promptly, skips are observable, and events delivered to different instances serialize under the **Thread Lock** without cross-instance supersession. This is exactly what main ships and documents on the strategy GoDoc today; this ADR promotes that documented honesty to the decided contract for the v0.x line. -What replaces it is decided here at the contract level — a fenced **Lock Takeover** — while the full dispatch-integration protocol is deliberately deferred to a dedicated follow-up design. This ADR's own review demonstrated why: the takeover protocol couples to every dispatch phase and strategy (prelude vs tail placement, acknowledgement budgets, per-strategy conflict-discovery points) and to ADR 0002's accepted rule that the **Thread Lock** is acquired in the prelude, never after acknowledgement. Pinning that protocol piecemeal here reproduces the churn that parked PR #53; it must be designed once, against the dispatch structure as a whole. +## Cross-instance coalescing (issue #50): rejected for now -The capability sketch consistent with the requirements below (illustrative; the follow-up design finalizes the shape — in particular, the conflict-time observation most likely folds into the acquisition primitive as acquire-or-observe rather than a separate read): +A full State-backed design was drafted and reviewed on this very PR: per-scope fencing tokens from a global monotonic sequence, waiter registration with derived TTLs, dispatch-time supersession checks, holder-bound lock takeover with idempotent reconciliation, and fleet capability gating — an optional-capability shape per the ADR 0009 precedent. -```go -// LockHolder identifies a Thread Lock holder for compare-and-take. Identity is -// universally the lowercase hex SHA-256 of the lease token: stable across -// ExtendLock refreshes, computable by any caller from a token it generated, -// and non-reversible — observation never exposes release capability. -type LockHolder struct { - Key string - Identity string -} +**It is rejected for now, on evidence.** Across roughly fifteen adversarial review rounds, every round found new P1-severity protocol holes in the prose specification, clustered entirely in the distributed protocol: waiter barging past parked deliveries, TTL-reset fence reordering, allocation-to-registration stall gaps, divergence between local and global ordering sources, rolling-upgrade capability gaps, acknowledgement-deadline collisions with coordination round-trips, reads returning stale absence after registration expiry, ambiguous takeover commits, and replacement leases expiring mid-reconciliation. Each hole was individually fixable — and each fix added mechanism for the next round to break. That is the signature of specifying a distributed coordination protocol in prose: natural-language review can find holes one at a time, but can never establish their absence. A protocol whose correctness argument cannot be checked mechanically is not a foundation this runtime will make cross-instance ordering promises on. -// LockForcer is an optional State capability enabling preemption. -// TakeLock atomically replaces the lock iff its current holder is still the -// observed one, installing the caller-generated replacement lease. -// taken=false means the holder changed or the lock is free — never an error. -type LockForcer interface { - TakeLock(ctx context.Context, observed LockHolder, replacement LockLease, ttl time.Duration) (bool, error) -} -``` +**The reopening bar.** Issue #50 stays open as future work, gated on all of: -**Binding requirements** — this ADR's contract for the follow-up design; nothing weaker ships: +1. A **formally modeled design** — the protocol (fences, registration lifetimes, takeover, degradation) specified and checked in a model checker (e.g. TLA+) against explicit invariants (no stale dispatch after a newer turn, no successor-lease invalidation, no accepted-event loss beyond the documented crash class) — or adoption of a **proven external primitive** that provides the ordering guarantee outright. +2. **Demonstrated user demand**: a real multi-instance deployment for which per-instance supersession plus **Thread Lock** serialization is insufficient in practice, not in principle. -1. **Takeover, not release-then-acquire.** A single compare-and-swap on the lock key: no window in which a third party can acquire between "release" and "acquire" (class 3 precluded by construction). A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must, and then the preemptor falls back to the configured strategy path, observed — no retry loop. -2. **Conflict-time binding, captured atomically.** The takeover may only present the holder identity captured *atomically with the failed acquisition that constituted the Lock Conflict* — an acquire-or-observe primitive, not a separate later read, which reintroduces the successor race (a victim finishing and a successor acquiring between conflict and observation). A preemptor must be structurally unable to take a successor's lease. -3. **Idempotent, renewing reconciliation.** The replacement lease token is caller-generated. An ambiguous takeover (committed, response lost) is reconciled by *extending* the replacement — confirming and renewing in one probe — never by observation alone, which cannot renew and lets the replacement expire mid-investigation; absence observed after the replacement's TTL horizon is never proof of non-commit. Inconclusive outcomes fall back with their own observation as a documented backend-outage residual. -4. **One path, no local registries.** Preemption goes through the State whether the victim is local or remote (classes 4 and 5 precluded: no process-map/State agreement problem, no victim-drain handshake). The victim stops via the universal lease-loss cancellation already on main, with every lease-refresh call independently time-bounded at one refresh interval (a refresh that cannot complete counts as lease loss — today's `ExtendLock` under `context.WithoutCancel` has no RPC deadline). Signal latency: at most one refresh interval plus the per-call bound (`ThreadLockTTL` total at defaults); termination is cooperative — Go cannot forcibly stop a handler — and a handler that ignores cancellation overlaps unboundedly, the same contract as `DetachTimeout`, shutdown drain, and lease loss. A local-victim nudge may only ever be best-effort latency sugar, never correctness machinery. -5. **The acknowledgement is never hostage.** Neither the `OnLockConflict` hook (application code) nor takeover/reconciliation latency may gate the platform acknowledgement. -6. **Fail-fast gating.** Preemption requires `DispatchDeferred` (a sync holder cannot be stopped), a State implementing `LockForcer` (constructor error otherwise — a preemption hook on a State that cannot preempt is a misconfiguration, not a degradation), and a strategy whose conflict point is well-defined for the hook: `ConcurrencyConcurrent` (no lock, no conflict) and `ConcurrencyBurst` (one lock for many members, one `*Event` in the hook) are rejected at construction. Uniform fleet configuration is an operator contract, as in §3. -7. **ADR 0002 reconciled explicitly.** ADR 0002 requires the Thread Lock acquired in the prelude and rejects post-acknowledgement acquisition. Any protocol that swaps or acquires the lock after acknowledgement reopens that rule and must supersede it explicitly in the follow-up design — never silently. +**What stands regardless of the future protocol:** -**Open questions the follow-up design must resolve before any preemption implementation:** the acquire-or-observe primitive's exact shape on the required-vs-optional interface boundary; the takeover's placement relative to ADR 0002's prelude-lock rule; the conflict-observation point for strategies that discover contention in the tail (debounce takes no prelude lock — define its observation point or reject debounce + preemption); and the interaction between takeover and the §3 fence check for conflicted preemptors. Until that design is accepted, the force/steerability surface stays reserved, exactly as on main today. - -### 5. Composition - -`Coalescer` and `LockForcer` are independent capabilities, and each alone is honest: coalescing without preemption gives global newest-wins with strategy-path conflict handling; preemption without coalescing gives fenced takeover with per-instance-only supersession (a stale preemptor may then dispatch its event — today's per-instance semantics, serialized as ever by the **Thread Lock**). Implementing both gives the full contract. The conformance suite tests each capability separately plus the composition. +- **Key-only `ForceReleaseLock(ctx, key)` is rejected finally** (ADR 0012's proposed force shape). The #38 history shows key-only identity cannot distinguish victim from successor ([r3871938987](https://github.com/coder/chat/pull/38#discussion_r3871938987)), and release-then-acquire leaves a gap a third party can enter. Any future force primitive must bind to the holder observed at conflict time, atomically. +- **Local preemption choreography is rejected finally**: the staged `inflightCancels` registry, `preemptLocalIfPending`, and the `victimDone` victim-drain handshake were the single largest P1 source on #38 (e.g. [r3872104899](https://github.com/coder/chat/pull/38#discussion_r3872104899), [r3871040671](https://github.com/coder/chat/pull/38#discussion_r3871040671)). +- **No event payloads in Runtime State**, whatever the protocol: coordination-only is a standing domain rule, so cross-instance waiter takeover (which requires dispatchable payloads in State) is out at any bar. +- The `burst` and force/steerability names remain reserved (ADR 0012); force/steerability is additionally gated on the same formal-design bar, since preemption is the same class of distributed protocol. ## Failure-mode disposition (the #38 anti-examples) -| # | Class | Disposition in this design | -|---|---|---| -| 1 | Unbounded admission | Precluded: pre-ack `MaxDetached` gate; per-scope `MaxBurstBatch` seal-and-roll | -| 2 | Process-local coordination, distributed semantics | Precluded: cross-instance semantics are only claimed where a State capability backs them; absence degrades to documented per-instance behavior; preemption is single-path via State | -| 3 | Key-only force release | Precluded: `TakeLock` compare-and-swap on observed holder identity; no release→acquire gap | -| 4 | Non-atomic handoff/registration races | Precluded: no local preemption registries to race with State updates; cross-instance ordering is enforced by State atomics (fence CAS, lock CAS) | -| 5 | Premature handover | Transformed and documented: no victim-drain handshake; the victim's context is cancelled within one refresh interval plus the bounded refresh call, termination is cooperative — overlap lasts the signal latency plus the victim's cancellation latency (unbounded only for handlers that ignore their context) | -| 6 | Lease-lifecycle divergence | Precluded by rule: all lock-holding paths (including burst) reuse the one refresh/cancel/outcome implementation | -| 7 | Temporal/budget skew | Precluded: early fence allocation pins cross-instance admission order; per-member execution budgets; no shared batch deadline | - -Accepted residuals, all observable: coalesced-turn loss on register-then-die (§3); preemption overlap of the cancellation-signal latency plus the victim's cooperative cancellation latency (§4, requirement 4); fence order approximating State-arrival order, not platform send order (§3); per-instance degradation for deliveries whose prelude stalls past the registration window (§3). Preemption's remaining residuals are finalized by §4's follow-up design within its binding requirements. +| Class | Disposition in this design | +|---|---| +| Unbounded admission | Precluded: pre-ack `MaxDetached` gate; per-scope `MaxBurstBatch` seal-and-roll | +| Process-local coordination with distributed semantics implied | Precluded by scope honesty: v0.x claims per-instance semantics only (§3); no distributed protocol ships without the formal bar | +| Key-only force release | Rejected finally; no force primitive ships in v0.x | +| Non-atomic ownership handoff; premature handover | Not shipped: no preemption, no local preemption registries in v0.x | +| Lease-lifecycle divergence | Precluded by rule: all lock-holding paths (including future burst) reuse the one refresh/cancel/outcome implementation | +| Temporal/budget skew | Precluded where in scope: per-member execution budgets, no shared batch deadline; local admission-sequence ordering already on main | ## Outcome for PR #53 (staged burst + preemption) -**Verdict: close PR #53.** Neither half should merge in its current shape, and the halves should not travel together again. +**Verdict: close PR #53.** Judged against per-instance semantics plus the admission bound: -- **Burst: rewrite on this design, salvaging the concept and its tests.** `ConcurrencyBurst` and per-member fresh budgets survive. Required changes: members count against `MaxDetached`; `MaxBurstBatch` with seal-and-roll; the batch lock lifecycle must reuse the shared refresh/cancel machinery (the staged branch's separate refresh loop reproduced class 6 twice). Ship as its own PR after the admission bound lands. -- **Preemption: close outright; revival is gated on §4's follow-up design.** Rejected finally: key-only `ForceReleaseLock(ctx, key)` and all four backend implementations of it; the `inflightCancels` registry; `preemptLocalIfPending`/`victimDone`. Surviving as reserved names under §4's binding requirements: `OnLockConflict` (deferred-only, `LockForcer`-required, concurrent/burst rejected), `ErrPreempted`/`OutcomePreempted` (already on main), and the `LockForcer` fenced-takeover capability. No preemption PR is acceptable until the dedicated protocol design resolves §4's open questions; it ships last, after the `Coalescer` fence exists. +- **Burst: revive with changes, as its own PR, after the admission bound lands.** Burst is per-instance batching and needs no cross-instance machinery. Required changes: members count against `MaxDetached`; `MaxBurstBatch` with seal-and-roll; per-member fresh `DetachTimeout` budgets; the batch lock lifecycle must reuse the shared refresh/cancel machinery (the staged branch's separate refresh loop reproduced the lease-lifecycle failure class twice); `DispatchDeferred` required at construction. +- **Preemption: rejected pending the formal-design bar.** Key-only `ForceReleaseLock` and the local preemption choreography are rejected finally (above); a safe replacement is a distributed protocol of exactly the class this ADR declines to specify in prose. `ErrPreempted`/`OutcomePreempted` (already on main, serving lease-loss cancellation) are unaffected. ## Non-goals This design explicitly refuses to promise: -- **Exactly-once (or at-least-once) cross-instance dispatch.** Coalescing is at-most-newest per group; deferred dispatch's crash-loss contract (ADR 0002) is inherited, and supersede-then-crash loses the group's turn. -- **Cross-instance waiter takeover.** It would require event payloads in **Runtime State** — a message store, refused on the coordination-only rule. -- **Cross-instance burst batch merging.** Same refusal; per-instance batches serialized by the **Thread Lock** are the contract. +- **Cross-instance supersession of any kind in v0.x** — see the rejection section; per-instance is the contract. +- **Admission control for synchronous dispatch.** Under `DispatchSync` the goroutine and payload exist at the HTTP layer before the runtime sees the delivery; a runtime cap cannot shed that load, and `net/http` imposes no request-concurrency limit of its own. Bounding synchronous serving is an explicit serving-layer operator contract. - **Byte-accounted admission.** The runtime cannot meaningfully measure retained `Raw` platform payloads plus handler closures; the bound is a count, and operators size it against their platform's payload ceiling. - **Fleet-wide admission control.** `MaxDetached` protects one instance's memory; fleet capacity management belongs to the operator's front door. -- **Zero-overlap — or even hard-bounded — preemption.** A strict victim-drain handoff is exactly the class-4/5 churn machine, and Go cannot forcibly stop a handler regardless: the contract is prompt cancellation and cooperative termination, nothing stronger. -- **Admission control for synchronous dispatch.** Under `DispatchSync` the goroutine and payload exist at the HTTP layer before the runtime sees the delivery; a runtime cap cannot shed that load, and `net/http` imposes no request-concurrency limit of its own. Bounding synchronous serving is an explicit serving-layer operator contract (request limiting in front of the **Webhook Handler**), not a runtime promise. -- **Cross-instance coalescing for synchronous dispatch.** A synchronous queue park is bounded only by the caller's request context, which need not carry a deadline, so no sound registration TTL exists for it. Sync queue keeps per-instance semantics; cross-instance coalescing pairs with `DispatchDeferred`, whose parks `DetachTimeout` bounds. -- **State watch/notify primitives.** Coalescing is check-at-dispatch; no backend is required to push notifications. +- **Exactly-once dispatch.** ADR 0002's crash-loss contract for deferred dispatch is inherited unchanged. ## Consequences - Two new **Runtime Options** (`MaxDetached`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. - Adapters gain one honesty duty: map `ErrAdmissionRejected` shape-awarely — a retry-inducing response for platform-redelivered shapes, a truthful busy response for direct invocations the platform does not redeliver. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. -- The `State` interface itself does not change; both extensions are optional capabilities (ADR 0009 precedent). All four in-repo States implement `Coalescer` when its integration ships; `LockForcer` implementations follow §4's follow-up design. The conformance suite grows capability sections (fence monotonicity, register-only-newer, TTL expiry; for takeover later: compare-and-swap atomicity under contention, extend-churn non-defeat). -- Queue/debounce dispatch under `DispatchDeferred` gains up to three State round-trips (allocate, register, check) when `Coalescer` is present — the cost of global newest-wins. Absent the capability, or under sync dispatch, nothing new is paid. -- ADR 0012's proposed force surface is superseded by §4 (flagged per the domain docs' ADR-conflict rule). ADR 0012's status note and the CONTEXT.md glossary (**Admission Bound**, **Waiter Fence**, **Lock Takeover**) update when this ADR is accepted and implementation lands. -- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) `Coalescer` + backends + conformance; (3) queue/debounce fence integration — closes #50; (4) burst revival; (5) the dedicated preemption-protocol design resolving §4's open questions within its binding requirements, then the fenced-preemption implementation (including the independently time-bounded lease-refresh calls). +- The `State` interface does not change, and no optional State capability ships with this ADR. +- Multi-instance deployments keep the documented per-instance coalescing semantics indefinitely, until the reopening bar is met. Operators who need stronger ordering today must route same-scope traffic to one instance (sticky routing) — an operational workaround, stated honestly, not a runtime promise. +- ADR 0012's proposed force surface (`ForceReleaseLock` by key) is superseded by the final rejection here (flagged per the domain docs' ADR-conflict rule); its `burst` reservation is unchanged and its implementation is unblocked by §2. +- The CONTEXT.md glossary gains **Admission Bound** when this ADR is accepted and implementation lands. +- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) burst revival per the PR #53 outcome. Issue #50 remains open, labeled as gated on the reopening bar. ## Alternatives Considered @@ -207,32 +125,12 @@ Rejected. Parking the webhook goroutine busts platform ack deadlines (Slack's 3s Rejected as the whole answer. A load balancer cannot see detached-tail occupancy — the exhausted resource is invisible outside the runtime. Front-door rate limiting remains complementary. -### Required `State` methods instead of optional capabilities +### Ship the cross-instance coalescing protocol now (the original scope of this ADR) -Rejected. Growing the required interface breaks every third-party State for features many deployments never enable; the `HistoryReader` precedent (ADR 0009) established capability discovery with honest degradation. +Rejected — this is the decision the rejection section records. The drafted protocol (fences, registration TTLs, fenced takeover, reconciliation) accumulated new P1-severity holes in every prose review round without converging; correctness of this protocol class must be established by model checking or a proven primitive, not by iterative prose review. ### Payload-bearing waiter registry in State (global coalescing with takeover) -Rejected on the coordination-only rule: **Runtime State** would become a message store, with the size limits, retention, and privacy surface the contract deliberately excludes. The fence carries ordering, never content. - -### Key-only `ForceReleaseLock` (ADR 0012's proposed shape, PR #53's implementation) - -Rejected. Key-only identity cannot distinguish victim from successor (class 3), and release-then-acquire reopens the race even with identity. Compare-and-take on a single key moves the atomicity into a State primitive every backend can actually provide. - -### Local preemption fast-path registry (`inflightCancels` + `victimDone`) - -Rejected as correctness machinery. Two coordination systems — process maps and the State — must agree at every interleaving, and the #38 history is the catalog of ways they didn't. One path, through the State; local nudges may only ever be best-effort latency sugar. - -### Cross-instance debounce timer coordination (a global quiet period) - -Rejected. A globally-reset quiet period needs watch/notify primitives or polling loops in every parked waiter. Check-at-dispatch delivers the observable contract — only the newest dispatches — without new State primitives; the quiet period stays a per-instance approximation. - -### Single-phase fence allocation (allocate at registration time) - -Rejected. Registration order diverges from arrival order when preludes stall on State round-trips: a stale delivery could fence out a newer waiter (class 7, cross-instance). Allocating before the reorderable work pins the order; registering only routed deliveries keeps duplicates and failures from superseding anyone. - -### Per-scope fence counters with TTL - -Rejected. A per-scope counter must either live forever — unique-scope traffic then grows Memory/Redis/Postgres coordination state permanently — or expire, and an expired-then-reset counter lets an old delivery's high fence outrank a fresh low one, reversing newest-wins exactly when a scope goes quiet. One global non-resetting sequence costs O(1) permanent state, preserves strict ordering across expiry of the per-scope register entries, and its per-scope subsequence is still strictly increasing. +Rejected on the coordination-only rule at any bar: **Runtime State** would become a message store, with the size limits, retention, and privacy surface the contract deliberately excludes. -Cross-references: ADR 0002 (deferred dispatch, crash-loss contract), ADR 0009 (optional-capability precedent), ADR 0012 (strategy set; force surface reshaped here), ADR 0014 (NATS State mechanics reused for fences and takeover); issues #44 and #50; the PR #38 review history and PR #53 (the staged branch this ADR disposes). +Cross-references: ADR 0002 (deferred dispatch, crash-loss contract), ADR 0009 (optional-capability precedent and the storage rule), ADR 0012 (strategy set; force surface superseded here), issues #44 and #50; the PR #38 review history and PR #53 (the staged branch this ADR disposes); the review history of this ADR's own PR (#54) as the rejection evidence. From f4dbb6464c41c6c3416916b455ec2f40dbc84875 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 19:18:13 +0000 Subject: [PATCH 17/28] docs(adr): prelude-return slot release, ADR 0003 acceptance qualification, explicit ADR 0012 supersessions for burst gating --- docs/adr/0015-runtime-coordination.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index f717db9..ac71021 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -19,13 +19,15 @@ Both were drafted here as one design. The admission bound converged under review ### 1. Admission Bound (issue #44) -Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot at acknowledgement; only routed deferred work holds it to the tail's end. +Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot **when the prelude returns** — the runtime's own boundary, before the adapter writes any response; an errored prelude is not acknowledged at all, and the runtime never observes the adapter's write, so prelude return is the only release point that cannot leak permits. Only routed deferred work holds its slot to the tail's terminal outcome. Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to a shape-aware response (adapter-owned). Because the delivery was never acknowledged and never marked, no dedupe record blocks a retry — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away). The mapping is shape-aware because platforms do not retry every webhook shape: - **Platform-retried deliveries** (e.g. Slack Events API callbacks): the adapter returns the retry-inducing response (HTTP 429/503 equivalents) and the platform's own redelivery covers the event. - **Direct user invocations that the platform does not redeliver** (e.g. Slack slash commands and interactivity — the in-repo Slack normalization records that these carry no retry headers): a bare 429 would turn a click into a silent permanent failure, so the adapter must answer with a *truthful busy response* to the user (a visible "busy, try again" acknowledgement, observed as rejected) — never a silent failure, never a fake success. The user's own retry is honored because no dedupe record exists. +This qualifies ADR 0003 explicitly (flagged per the domain docs' ADR-conflict rule): ADR 0003 defines a verified, normalized command as an **Accepted Event** owed acknowledgement, but the admission gate precedes acceptance — a delivery rejected at admission never becomes an **Accepted Event**, is never marked, and owes only the shape-aware overload response above. Overload rejection is a pre-acceptance outcome, not a broken acknowledgement of an accepted command. + Placement and validation: - The gate sits at the head of the prelude, before any **Runtime State** write, so a rejected delivery leaves no record. @@ -107,7 +109,7 @@ This design explicitly refuses to promise: - Adapters gain one honesty duty: map `ErrAdmissionRejected` shape-awarely — a retry-inducing response for platform-redelivered shapes, a truthful busy response for direct invocations the platform does not redeliver. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. - The `State` interface does not change, and no optional State capability ships with this ADR. - Multi-instance deployments keep the documented per-instance coalescing semantics indefinitely, until the reopening bar is met. Operators who need stronger ordering today must route same-scope traffic to one instance (sticky routing) — an operational workaround, stated honestly, not a runtime promise. -- ADR 0012's proposed force surface (`ForceReleaseLock` by key) is superseded by the final rejection here (flagged per the domain docs' ADR-conflict rule); its `burst` reservation is unchanged and its implementation is unblocked by §2. +- Three ADR 0012 statements are superseded here (flagged per the domain docs' ADR-conflict rule): its proposed force surface (`ForceReleaseLock` by key) falls to the final rejection above; its staging gate that held `burst` on "fenced-coordination design work" is lifted — burst ships per-instance under §2, gated only on the admission bound; and its expectation that `queue`/`debounce`/`burst` "need wait/coalesce coordination" expanding every State implementation is withdrawn for v0.x — per-instance supersession (§3) needs no State expansion, and any future coordination contract goes through the reopening bar. ADR 0003 is qualified as described in §1. - The CONTEXT.md glossary gains **Admission Bound** when this ADR is accepted and implementation lands. - Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) burst revival per the PR #53 outcome. Issue #50 remains open, labeled as gated on the reopening bar. From 559a7f7e46692d2e3e66fa171cbc96c21ee4bdd2 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 19:28:54 +0000 Subject: [PATCH 18/28] docs(adr): slot release at tail-goroutine return, duplicate fast path at the cap, optional per-tenant sublimit --- docs/adr/0015-runtime-coordination.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index ac71021..24cbc00 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -19,7 +19,7 @@ Both were drafted here as one design. The admission bound converged under review ### 1. Admission Bound (issue #44) -Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot **when the prelude returns** — the runtime's own boundary, before the adapter writes any response; an errored prelude is not acknowledged at all, and the runtime never observes the adapter's write, so prelude return is the only release point that cannot leak permits. Only routed deferred work holds its slot to the tail's terminal outcome. +Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot **when the prelude returns** — the runtime's own boundary, before the adapter writes any response; an errored prelude is not acknowledged at all, and the runtime never observes the adapter's write, so prelude return is the only release point that cannot leak permits. Only routed deferred work holds its slot, and it holds it until the detached tail goroutine actually returns — after handler completion *and* lock cleanup — not merely until a terminal outcome is recorded: cleanup can stall (today's release path runs under an uncancellable context after the outcome is recorded), and a slot released before the goroutine, event, and closure are actually gone would let sustained traffic exceed the cap by the amount of work stuck in cleanup. Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to a shape-aware response (adapter-owned). Because the delivery was never acknowledged and never marked, no dedupe record blocks a retry — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away). The mapping is shape-aware because platforms do not retry every webhook shape: @@ -30,9 +30,10 @@ This qualifies ADR 0003 explicitly (flagged per the domain docs' ADR-conflict ru Placement and validation: -- The gate sits at the head of the prelude, before any **Runtime State** write, so a rejected delivery leaves no record. +- The gate sits at the head of the prelude, before any **Runtime State** write, so a rejected delivery leaves no record. One read-only exception preserves ADR 0002's duplicate contract under saturation: at the cap, the gate checks **Event Identity** dedupe (a read, never a write) and acknowledges a known duplicate as a duplicate instead of rejecting it — a redelivery of an already-marked event consumes no detached capacity, and bouncing it would amplify platform retries exactly when the instance is saturated. - `MaxDetached` must be positive under `DispatchDeferred` (constructor validation, the `DetachTimeout` precedent); `DefaultRuntimeOptions` gains a default (1024). `DispatchSync` ignores it: a synchronous delivery's goroutine and payload belong to the HTTP request before dispatch begins, so a runtime-side cap cannot shed them — and `net/http` imposes no request-concurrency limit by itself. Bounding synchronous serving is therefore an explicit operator contract at the HTTP layer (a request/connection limiter in front of the **Webhook Handler**), stated in the option's GoDoc rather than assumed — see Non-goals. -- Observability: a rejected delivery emits a new observation (`admission_rejected`) and closes its dispatch span with a new terminal outcome (`admission-rejected`). Rejections are never silent. +- Observability: a rejected delivery emits a new observation (`admission_rejected`, carrying the adapter and tenant attributes) and closes its dispatch span with a new terminal outcome (`admission-rejected`). Rejections are never silent. +- Multi-tenant fairness: `MaxDetached` alone is a global ceiling, so on a multi-tenant runtime (ADR 0006) one tenant's sustained valid traffic could starve co-located tenants. An optional `MaxDetachedPerTenant int` (**Runtime Options**, 0 = disabled) bounds any single tenant's share of the admitted work using the same rejection path and observability; it is a sublimit, not a reservation — fair-share scheduling is out of scope. Deployments serving a single tenant need only the global bound. Interaction with each strategy (under `DispatchDeferred`): @@ -101,11 +102,12 @@ This design explicitly refuses to promise: - **Admission control for synchronous dispatch.** Under `DispatchSync` the goroutine and payload exist at the HTTP layer before the runtime sees the delivery; a runtime cap cannot shed that load, and `net/http` imposes no request-concurrency limit of its own. Bounding synchronous serving is an explicit serving-layer operator contract. - **Byte-accounted admission.** The runtime cannot meaningfully measure retained `Raw` platform payloads plus handler closures; the bound is a count, and operators size it against their platform's payload ceiling. - **Fleet-wide admission control.** `MaxDetached` protects one instance's memory; fleet capacity management belongs to the operator's front door. +- **Fair-share tenant scheduling.** `MaxDetachedPerTenant` is a per-tenant ceiling, not a reservation or weighted scheduler; guaranteed tenant throughput under saturation is not promised. - **Exactly-once dispatch.** ADR 0002's crash-loss contract for deferred dispatch is inherited unchanged. ## Consequences -- Two new **Runtime Options** (`MaxDetached`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. +- Three new **Runtime Options** (`MaxDetached`, optional `MaxDetachedPerTenant`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. - Adapters gain one honesty duty: map `ErrAdmissionRejected` shape-awarely — a retry-inducing response for platform-redelivered shapes, a truthful busy response for direct invocations the platform does not redeliver. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. - The `State` interface does not change, and no optional State capability ships with this ADR. - Multi-instance deployments keep the documented per-instance coalescing semantics indefinitely, until the reopening bar is met. Operators who need stronger ordering today must route same-scope traffic to one instance (sticky routing) — an operational workaround, stated honestly, not a runtime promise. From b9d8da5de750b2bfe0aa8b35de0eecf3d3b7faa5 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 19:40:36 +0000 Subject: [PATCH 19/28] docs(adr): drop unimplementable duplicate fast path (qualify ADR 0002, prohibit mark-as-probe), composite tenant key + empty-tenant bucket + validation, burst final-member slot through cleanup --- docs/adr/0015-runtime-coordination.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 24cbc00..a02d66c 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -30,10 +30,10 @@ This qualifies ADR 0003 explicitly (flagged per the domain docs' ADR-conflict ru Placement and validation: -- The gate sits at the head of the prelude, before any **Runtime State** write, so a rejected delivery leaves no record. One read-only exception preserves ADR 0002's duplicate contract under saturation: at the cap, the gate checks **Event Identity** dedupe (a read, never a write) and acknowledges a known duplicate as a duplicate instead of rejecting it — a redelivery of an already-marked event consumes no detached capacity, and bouncing it would amplify platform retries exactly when the instance is saturated. +- The gate sits at the head of the prelude, before any **Runtime State** interaction, so a rejected delivery leaves no record. A consequence, stated as an explicit qualification of ADR 0002's duplicate contract (per the domain docs' ADR-conflict rule): the required `State` contract has no read-only **Event Identity** check (`MarkEvent` writes on a miss), so at the cap the gate cannot distinguish a redelivery of an already-marked event — under saturation, duplicates receive the overload response like any other delivery and converge to the ordinary duplicate acknowledgement once capacity frees. Probing dedupe with `MarkEvent` is prohibited: marking a first delivery before rejecting it would let the platform's retry be acknowledged as a duplicate and never handled — silent event loss, strictly worse than extra retries. If saturation retry amplification proves material in practice, a read-only dedupe check can be added later as a small optional State capability; it is deliberately not part of this ADR. - `MaxDetached` must be positive under `DispatchDeferred` (constructor validation, the `DetachTimeout` precedent); `DefaultRuntimeOptions` gains a default (1024). `DispatchSync` ignores it: a synchronous delivery's goroutine and payload belong to the HTTP request before dispatch begins, so a runtime-side cap cannot shed them — and `net/http` imposes no request-concurrency limit by itself. Bounding synchronous serving is therefore an explicit operator contract at the HTTP layer (a request/connection limiter in front of the **Webhook Handler**), stated in the option's GoDoc rather than assumed — see Non-goals. - Observability: a rejected delivery emits a new observation (`admission_rejected`, carrying the adapter and tenant attributes) and closes its dispatch span with a new terminal outcome (`admission-rejected`). Rejections are never silent. -- Multi-tenant fairness: `MaxDetached` alone is a global ceiling, so on a multi-tenant runtime (ADR 0006) one tenant's sustained valid traffic could starve co-located tenants. An optional `MaxDetachedPerTenant int` (**Runtime Options**, 0 = disabled) bounds any single tenant's share of the admitted work using the same rejection path and observability; it is a sublimit, not a reservation — fair-share scheduling is out of scope. Deployments serving a single tenant need only the global bound. +- Multi-tenant fairness: `MaxDetached` alone is a global ceiling, so on a multi-tenant runtime (ADR 0006) one tenant's sustained valid traffic could starve co-located tenants. An optional `MaxDetachedPerTenant int` (**Runtime Options**, 0 = disabled, negative rejected at **Runtime Construction**) bounds any single installation's share of the admitted work using the same rejection path and observability. Accounting keys on the composite `(adapter, tenant)` — ADR 0006's installation identity — never the bare tenant string, so same-named tenants on different adapters do not share a bucket; an empty tenant is counted as that adapter's single untenanted bucket (it must not bypass the limit), meaning per-installation isolation is only meaningful for adapters that populate `Event.Tenant`. It is a sublimit, not a reservation — fair-share scheduling is out of scope. Deployments serving a single tenant need only the global bound. Interaction with each strategy (under `DispatchDeferred`): @@ -49,10 +49,10 @@ Interaction with each strategy (under `DispatchDeferred`): Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. When burst ships (see the PR #53 outcome), batch growth is bounded twice: -- **Globally** by `MaxDetached`: each member counts from admission to its member-run's terminal outcome. +- **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). - **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) -Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Byte-based accounting is refused — see Non-goals. +Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end, except the batch's shared tail goroutine is itself covered — the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. ### 3. Per-instance supersession is the v0.x contract From f5dd0ae5ec9b526153d736ee45845222cfdf9254 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 19:49:16 +0000 Subject: [PATCH 20/28] docs(adr): consistent slot-lifetime wording; burst lock sequencing per shipped debounce precedent with explicit ADR 0002 qualification --- docs/adr/0015-runtime-coordination.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index a02d66c..2da0e66 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -19,7 +19,7 @@ Both were drafted here as one design. The admission bound converged under review ### 1. Admission Bound (issue #44) -Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot **when the prelude returns** — the runtime's own boundary, before the adapter writes any response; an errored prelude is not acknowledged at all, and the runtime never observes the adapter's write, so prelude return is the only release point that cannot leak permits. Only routed deferred work holds its slot, and it holds it until the detached tail goroutine actually returns — after handler completion *and* lock cleanup — not merely until a terminal outcome is recorded: cleanup can stall (today's release path runs under an uncancellable context after the outcome is recorded), and a slot released before the goroutine, event, and closure are actually gone would let sustained traffic exceed the cap by the amount of work stuck in cleanup. +Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the slot-release point defined next (not until a terminal outcome is *recorded*; recording precedes cleanup). Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot **when the prelude returns** — the runtime's own boundary, before the adapter writes any response; an errored prelude is not acknowledged at all, and the runtime never observes the adapter's write, so prelude return is the only release point that cannot leak permits. Only routed deferred work holds its slot, and it holds it until the detached tail goroutine actually returns — after handler completion *and* lock cleanup — not merely until a terminal outcome is recorded: cleanup can stall (today's release path runs under an uncancellable context after the outcome is recorded), and a slot released before the goroutine, event, and closure are actually gone would let sustained traffic exceed the cap by the amount of work stuck in cleanup. Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to a shape-aware response (adapter-owned). Because the delivery was never acknowledged and never marked, no dedupe record blocks a retry — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away). The mapping is shape-aware because platforms do not retry every webhook shape: @@ -47,7 +47,7 @@ Interaction with each strategy (under `DispatchDeferred`): ### 2. Burst shaping (issue #44's scope extension) -Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. When burst ships (see the PR #53 outcome), batch growth is bounded twice: +Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail), flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). - **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) From 8d0c4aeef7487204d05d9a27eaf57d40bcd527dc Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 19:58:03 +0000 Subject: [PATCH 21/28] docs(adr): batch FIFO seal-order dispatch, batch-derived member retention bound (ADR 0002 qualification), budgeted busy-response for ack-separated interactions --- docs/adr/0015-runtime-coordination.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 2da0e66..ce1424e 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -28,6 +28,8 @@ Semantics at the cap: **reject-with-signal, before ack and before dedupe marking This qualifies ADR 0003 explicitly (flagged per the domain docs' ADR-conflict rule): ADR 0003 defines a verified, normalized command as an **Accepted Event** owed acknowledgement, but the admission gate precedes acceptance — a delivery rejected at admission never becomes an **Accepted Event**, is never marked, and owes only the shape-aware overload response above. Overload rejection is a pre-acceptance outcome, not a broken acknowledgement of an accepted command. +For interaction shapes whose platform contract separates the acknowledgement from the user-visible response (ADR 0004: Slack `block_actions` requires an empty 2xx, with messages via `response_url`), the busy response cannot ride the acknowledgement body. The adapter acknowledges promptly per ADR 0004 and sends the busy message through the follow-up channel as a single, short-deadline, bounded adapter call under a small fixed busy-response budget — not a detached tail, so it does not consume `MaxDetached` capacity, and the budget caps its goroutines under saturation. If even that budget is exhausted, the busy post is skipped: the rejection is still observed (`admission_rejected`), degrading only the user-visible signal, never the acknowledgement — the one narrow, budget-bounded exception to the visible-busy-response rule, stated honestly. + Placement and validation: - The gate sits at the head of the prelude, before any **Runtime State** interaction, so a rejected delivery leaves no record. A consequence, stated as an explicit qualification of ADR 0002's duplicate contract (per the domain docs' ADR-conflict rule): the required `State` contract has no read-only **Event Identity** check (`MarkEvent` writes on a miss), so at the cap the gate cannot distinguish a redelivery of an already-marked event — under saturation, duplicates receive the overload response like any other delivery and converge to the ordinary duplicate acknowledgement once capacity frees. Probing dedupe with `MarkEvent` is prohibited: marking a first delivery before rejecting it would let the platform's retry be acknowledged as a duplicate and never handled — silent event loss, strictly worse than extra retries. If saturation retry amplification proves material in practice, a read-only dedupe check can be added later as a small optional State capability; it is deliberately not part of this ADR. @@ -50,7 +52,9 @@ Interaction with each strategy (under `DispatchDeferred`): Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail), flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). -- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) +- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. + +A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO) — not by one `DetachTimeout`. Operators size `MaxBurstBatch` and `DetachTimeout` together with that product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end, except the batch's shared tail goroutine is itself covered — the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. From f900f11389a7fa0523eae74d8ec6a60b68a936d9 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 20:08:50 +0000 Subject: [PATCH 22/28] docs(adr): burst window = DebounceInterval, bounded batch coordination with observable abandonment, lease-loss batch disposition --- docs/adr/0015-runtime-coordination.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index ce1424e..2d1802f 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -52,9 +52,9 @@ Interaction with each strategy (under `DispatchDeferred`): Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail), flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). -- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. +- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst exactly as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. -A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO) — not by one `DetachTimeout`. Operators size `MaxBurstBatch` and `DetachTimeout` together with that product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. +A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, as on main) and the batch terminates — remaining members close observably with the skipped-on-lease-loss outcome, never run without the **Thread Lock**, and are never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The only unbounded residual is the §1 stalled-cleanup case, which is exactly why slots count until goroutine return rather than outcome recording. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end, except the batch's shared tail goroutine is itself covered — the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. From 88e12059eb3f1b4f70334ce8909306c69b0a0ab1 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 20:17:19 +0000 Subject: [PATCH 23/28] docs(adr): cap-reaching member ownership, cooperative-bound residual, no-response-url fallback, no-undocumented-loss invariant --- docs/adr/0015-runtime-coordination.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 2d1802f..c316e9d 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -28,7 +28,7 @@ Semantics at the cap: **reject-with-signal, before ack and before dedupe marking This qualifies ADR 0003 explicitly (flagged per the domain docs' ADR-conflict rule): ADR 0003 defines a verified, normalized command as an **Accepted Event** owed acknowledgement, but the admission gate precedes acceptance — a delivery rejected at admission never becomes an **Accepted Event**, is never marked, and owes only the shape-aware overload response above. Overload rejection is a pre-acceptance outcome, not a broken acknowledgement of an accepted command. -For interaction shapes whose platform contract separates the acknowledgement from the user-visible response (ADR 0004: Slack `block_actions` requires an empty 2xx, with messages via `response_url`), the busy response cannot ride the acknowledgement body. The adapter acknowledges promptly per ADR 0004 and sends the busy message through the follow-up channel as a single, short-deadline, bounded adapter call under a small fixed busy-response budget — not a detached tail, so it does not consume `MaxDetached` capacity, and the budget caps its goroutines under saturation. If even that budget is exhausted, the busy post is skipped: the rejection is still observed (`admission_rejected`), degrading only the user-visible signal, never the acknowledgement — the one narrow, budget-bounded exception to the visible-busy-response rule, stated honestly. +For interaction shapes whose platform contract separates the acknowledgement from the user-visible response (ADR 0004: Slack `block_actions` requires an empty 2xx, with messages via `response_url`), the busy response cannot ride the acknowledgement body. The adapter acknowledges promptly per ADR 0004 and sends the busy message through the follow-up channel as a single, short-deadline, bounded adapter call under a small fixed busy-response budget — not a detached tail, so it does not consume `MaxDetached` capacity, and the budget caps its goroutines under saturation. If that budget is exhausted — or the interaction carries no follow-up channel at all (ADR 0004 treats `response_url` as optional) — the busy post is skipped: the rejection is still observed (`admission_rejected`), degrading only the user-visible signal, never the acknowledgement. These are the two narrow, honestly-stated exceptions to the visible-busy-response rule. Placement and validation: @@ -52,9 +52,9 @@ Interaction with each strategy (under `DispatchDeferred`): Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail), flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). -- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): when a scope's open window reaches the cap, the window **seals and rolls** — the sealed batch proceeds to dispatch and the incoming event opens the next window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst exactly as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. +- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): the incoming event is appended first, and a window that thereby reaches exactly `MaxBurstBatch` members **seals and rolls** — the sealed batch (cap-reaching member included) proceeds to dispatch, and the *next* event opens a new window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst exactly as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. -A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, as on main) and the batch terminates — remaining members close observably with the skipped-on-lease-loss outcome, never run without the **Thread Lock**, and are never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The only unbounded residual is the §1 stalled-cleanup case, which is exactly why slots count until goroutine return rather than outcome recording. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. +A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, as on main) and the batch terminates — remaining members close observably with the skipped-on-lease-loss outcome, never run without the **Thread Lock**, and are never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end, except the batch's shared tail goroutine is itself covered — the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. @@ -70,7 +70,7 @@ A full State-backed design was drafted and reviewed on this very PR: per-scope f **The reopening bar.** Issue #50 stays open as future work, gated on all of: -1. A **formally modeled design** — the protocol (fences, registration lifetimes, takeover, degradation) specified and checked in a model checker (e.g. TLA+) against explicit invariants (no stale dispatch after a newer turn, no successor-lease invalidation, no accepted-event loss beyond the documented crash class) — or adoption of a **proven external primitive** that provides the ordering guarantee outright. +1. A **formally modeled design** — the protocol (fences, registration lifetimes, takeover, degradation) specified and checked in a model checker (e.g. TLA+) against explicit invariants (no stale dispatch after a newer turn, no successor-lease invalidation, no *undocumented* **Accepted Event** loss — observable supersession and abandonment are the strategies' documented behavior and explicitly allowed; only the documented crash class may lose an event silently) — or adoption of a **proven external primitive** that provides the ordering guarantee outright. 2. **Demonstrated user demand**: a real multi-instance deployment for which per-instance supersession plus **Thread Lock** serialization is insufficient in practice, not in principle. **What stands regardless of the future protocol:** From ee38be80f6e80477c0953bb614653c5fa73ddc69 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 20:27:00 +0000 Subject: [PATCH 24/28] docs(adr): fixed burst window anchor, OutcomeSkippedLeaseLoss closed-set value, member reference clearing before slot release --- docs/adr/0015-runtime-coordination.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index c316e9d..0fb6b6e 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -52,11 +52,11 @@ Interaction with each strategy (under `DispatchDeferred`): Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail), flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). -- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): the incoming event is appended first, and a window that thereby reaches exactly `MaxBurstBatch` members **seals and rolls** — the sealed batch (cap-reaching member included) proceeds to dispatch, and the *next* event opens a new window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst exactly as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. +- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): the incoming event is appended first, and a window that thereby reaches exactly `MaxBurstBatch` members **seals and rolls** — the sealed batch (cap-reaching member included) proceeds to dispatch, and the *next* event opens a new window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. The timer is **fixed, anchored at the window's first member**: later arrivals join the window but never reset it — resetting is debounce's semantic and would park the first accepted member indefinitely under steady sub-cap traffic — so every batch dispatches within one `DebounceInterval` of its first member unless the size cap seals it sooner. -A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, as on main) and the batch terminates — remaining members close observably with the skipped-on-lease-loss outcome, never run without the **Thread Lock**, and are never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. +A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, closing with the existing `OutcomePreempted`) and the batch terminates — remaining members that never ran close with a new closed-set terminal outcome, `skipped-lease-loss` (`OutcomeSkippedLeaseLoss`), never `OutcomePreempted` (they were not running) and never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. -Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end, except the batch's shared tail goroutine is itself covered — the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. +Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end — and a completed member's event and closure references must be cleared from the batch's backing storage *before* its slot releases, so a released permit never has a still-reachable payload behind it (otherwise up to `MaxBurstBatch − 1` completed payloads per active batch would remain live beyond the cap) — except the batch's shared tail goroutine is itself covered: the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. ### 3. Per-instance supersession is the v0.x contract @@ -111,7 +111,7 @@ This design explicitly refuses to promise: ## Consequences -- Three new **Runtime Options** (`MaxDetached`, optional `MaxDetachedPerTenant`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. +- Three new **Runtime Options** (`MaxDetached`, optional `MaxDetachedPerTenant`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission (plus, when burst ships, the `skipped-lease-loss` terminal outcome). `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. - Adapters gain one honesty duty: map `ErrAdmissionRejected` shape-awarely — a retry-inducing response for platform-redelivered shapes, a truthful busy response for direct invocations the platform does not redeliver. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. - The `State` interface does not change, and no optional State capability ships with this ADR. - Multi-instance deployments keep the documented per-instance coalescing semantics indefinitely, until the reopening bar is met. Operators who need stronger ordering today must route same-scope traffic to one instance (sticky routing) — an operational workaround, stated honestly, not a runtime promise. From 43b658bdd20b1f0454e0b4ce8fc6b9a63df3aa09 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 20:38:49 +0000 Subject: [PATCH 25/28] docs(adr): shutdown drains open windows, handler-error continuation, detached busy-post context, index ADR 0015 in explanation.md --- docs/adr/0015-runtime-coordination.md | 4 ++-- docs/explanation.md | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 0fb6b6e..e74dbe3 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -28,7 +28,7 @@ Semantics at the cap: **reject-with-signal, before ack and before dedupe marking This qualifies ADR 0003 explicitly (flagged per the domain docs' ADR-conflict rule): ADR 0003 defines a verified, normalized command as an **Accepted Event** owed acknowledgement, but the admission gate precedes acceptance — a delivery rejected at admission never becomes an **Accepted Event**, is never marked, and owes only the shape-aware overload response above. Overload rejection is a pre-acceptance outcome, not a broken acknowledgement of an accepted command. -For interaction shapes whose platform contract separates the acknowledgement from the user-visible response (ADR 0004: Slack `block_actions` requires an empty 2xx, with messages via `response_url`), the busy response cannot ride the acknowledgement body. The adapter acknowledges promptly per ADR 0004 and sends the busy message through the follow-up channel as a single, short-deadline, bounded adapter call under a small fixed busy-response budget — not a detached tail, so it does not consume `MaxDetached` capacity, and the budget caps its goroutines under saturation. If that budget is exhausted — or the interaction carries no follow-up channel at all (ADR 0004 treats `response_url` as optional) — the busy post is skipped: the rejection is still observed (`admission_rejected`), degrading only the user-visible signal, never the acknowledgement. These are the two narrow, honestly-stated exceptions to the visible-busy-response rule. +For interaction shapes whose platform contract separates the acknowledgement from the user-visible response (ADR 0004: Slack `block_actions` requires an empty 2xx, with messages via `response_url`), the busy response cannot ride the acknowledgement body. The adapter acknowledges promptly per ADR 0004 and sends the busy message through the follow-up channel as a single, short-deadline, bounded adapter call under a small fixed busy-response budget — not a detached tail, so it does not consume `MaxDetached` capacity, and the budget caps its goroutines under saturation. The call runs on its own short-deadline context detached from the request context (which the platform cancels when the acknowledgement returns) and bounded by **Runtime Shutdown** — never on `r.Context()`, which would cancel the post, and never synchronously before the acknowledgement, which would delay it. If that budget is exhausted — or the interaction carries no follow-up channel at all (ADR 0004 treats `response_url` as optional) — the busy post is skipped: the rejection is still observed (`admission_rejected`), degrading only the user-visible signal, never the acknowledgement. These are the two narrow, honestly-stated exceptions to the visible-busy-response rule. Placement and validation: @@ -54,7 +54,7 @@ Burst requires `DispatchDeferred` (constructor validation, the debounce preceden - **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). - **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): the incoming event is appended first, and a window that thereby reaches exactly `MaxBurstBatch` members **seals and rolls** — the sealed batch (cap-reaching member included) proceeds to dispatch, and the *next* event opens a new window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. The timer is **fixed, anchored at the window's first member**: later arrivals join the window but never reset it — resetting is debounce's semantic and would park the first accepted member indefinitely under steady sub-cap traffic — so every batch dispatches within one `DebounceInterval` of its first member unless the size cap seals it sooner. -A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, closing with the existing `OutcomePreempted`) and the batch terminates — remaining members that never ran close with a new closed-set terminal outcome, `skipped-lease-loss` (`OutcomeSkippedLeaseLoss`), never `OutcomePreempted` (they were not running) and never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. +A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. **Runtime Shutdown** must seal any open window and drain it like every other detached tail: window timers are cancelled, and parked members close observably (the existing shutdown-drain semantics) before **Runtime State** cleanup — an open window's acknowledged members hold admission slots and must never lose their terminal outcomes to a timer firing after State shutdown. An ordinary handler error does *not* terminate the batch: the failing member closes with the existing error outcome (observed, as for any handler error) and the batch continues with its remaining members — only lease loss and coordination abandonment end a batch early, preserving the delivery guarantee. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, closing with the existing `OutcomePreempted`) and the batch terminates — remaining members that never ran close with a new closed-set terminal outcome, `skipped-lease-loss` (`OutcomeSkippedLeaseLoss`), never `OutcomePreempted` (they were not running) and never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end — and a completed member's event and closure references must be cleared from the batch's backing storage *before* its slot releases, so a released permit never has a still-reachable payload behind it (otherwise up to `MaxBurstBatch − 1` completed payloads per active batch would remain live beyond the cap) — except the batch's shared tail goroutine is itself covered: the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. diff --git a/docs/explanation.md b/docs/explanation.md index 08f318a..c89839f 100644 --- a/docs/explanation.md +++ b/docs/explanation.md @@ -40,6 +40,7 @@ design deliberately diverges from Vercel Chat SDK. | [0012](adr/0012-concurrency-strategy.md) | Concurrency strategy expansion (`drop`/`queue`/`debounce`/`concurrent` + lock scope implemented; `burst` and force/steerability staged) | Accepted (staged) | | [0013](adr/0013-linear-generic-comments.md) | Linear generic issue/comment participation | Accepted | | [0014](adr/0014-nats-state-adapter.md) | NATS JetStream state adapter | Accepted | +| [0015](adr/0015-runtime-coordination.md) | Deferred-dispatch admission bound; cross-instance coalescing rejected for now | Proposed | ## The Short Version From bfe2afb95e6d05916350bc0f97f9d39249790b10 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 20:47:20 +0000 Subject: [PATCH 26/28] docs(adr): ADR 0004 burst lock qualification, atomic shutdown admission close, tenant counter cleanup --- docs/adr/0015-runtime-coordination.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index e74dbe3..3988655 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -35,7 +35,7 @@ Placement and validation: - The gate sits at the head of the prelude, before any **Runtime State** interaction, so a rejected delivery leaves no record. A consequence, stated as an explicit qualification of ADR 0002's duplicate contract (per the domain docs' ADR-conflict rule): the required `State` contract has no read-only **Event Identity** check (`MarkEvent` writes on a miss), so at the cap the gate cannot distinguish a redelivery of an already-marked event — under saturation, duplicates receive the overload response like any other delivery and converge to the ordinary duplicate acknowledgement once capacity frees. Probing dedupe with `MarkEvent` is prohibited: marking a first delivery before rejecting it would let the platform's retry be acknowledged as a duplicate and never handled — silent event loss, strictly worse than extra retries. If saturation retry amplification proves material in practice, a read-only dedupe check can be added later as a small optional State capability; it is deliberately not part of this ADR. - `MaxDetached` must be positive under `DispatchDeferred` (constructor validation, the `DetachTimeout` precedent); `DefaultRuntimeOptions` gains a default (1024). `DispatchSync` ignores it: a synchronous delivery's goroutine and payload belong to the HTTP request before dispatch begins, so a runtime-side cap cannot shed them — and `net/http` imposes no request-concurrency limit by itself. Bounding synchronous serving is therefore an explicit operator contract at the HTTP layer (a request/connection limiter in front of the **Webhook Handler**), stated in the option's GoDoc rather than assumed — see Non-goals. - Observability: a rejected delivery emits a new observation (`admission_rejected`, carrying the adapter and tenant attributes) and closes its dispatch span with a new terminal outcome (`admission-rejected`). Rejections are never silent. -- Multi-tenant fairness: `MaxDetached` alone is a global ceiling, so on a multi-tenant runtime (ADR 0006) one tenant's sustained valid traffic could starve co-located tenants. An optional `MaxDetachedPerTenant int` (**Runtime Options**, 0 = disabled, negative rejected at **Runtime Construction**) bounds any single installation's share of the admitted work using the same rejection path and observability. Accounting keys on the composite `(adapter, tenant)` — ADR 0006's installation identity — never the bare tenant string, so same-named tenants on different adapters do not share a bucket; an empty tenant is counted as that adapter's single untenanted bucket (it must not bypass the limit), meaning per-installation isolation is only meaningful for adapters that populate `Event.Tenant`. It is a sublimit, not a reservation — fair-share scheduling is out of scope. Deployments serving a single tenant need only the global bound. +- Multi-tenant fairness: `MaxDetached` alone is a global ceiling, so on a multi-tenant runtime (ADR 0006) one tenant's sustained valid traffic could starve co-located tenants. An optional `MaxDetachedPerTenant int` (**Runtime Options**, 0 = disabled, negative rejected at **Runtime Construction**) bounds any single installation's share of the admitted work using the same rejection path and observability. Accounting keys on the composite `(adapter, tenant)` — ADR 0006's installation identity — never the bare tenant string, so same-named tenants on different adapters do not share a bucket; an empty tenant is counted as that adapter's single untenanted bucket (it must not bypass the limit), meaning per-installation isolation is only meaningful for adapters that populate `Event.Tenant`. A counter entry is removed when its installation's last slot releases — tenant-churn traffic must not grow accounting state without bound. It is a sublimit, not a reservation — fair-share scheduling is out of scope. Deployments serving a single tenant need only the global bound. Interaction with each strategy (under `DispatchDeferred`): @@ -49,12 +49,12 @@ Interaction with each strategy (under `DispatchDeferred`): ### 2. Burst shaping (issue #44's scope extension) -Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail), flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: +Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail) — and, identically, of ADR 0004's requirement that deferred interactions acquire the **Lock Lease** before acknowledgement, which likewise assumed the immediate-dispatch strategies — both flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). - **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): the incoming event is appended first, and a window that thereby reaches exactly `MaxBurstBatch` members **seals and rolls** — the sealed batch (cap-reaching member included) proceeds to dispatch, and the *next* event opens a new window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. The timer is **fixed, anchored at the window's first member**: later arrivals join the window but never reset it — resetting is debounce's semantic and would park the first accepted member indefinitely under steady sub-cap traffic — so every batch dispatches within one `DebounceInterval` of its first member unless the size cap seals it sooner. -A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. **Runtime Shutdown** must seal any open window and drain it like every other detached tail: window timers are cancelled, and parked members close observably (the existing shutdown-drain semantics) before **Runtime State** cleanup — an open window's acknowledged members hold admission slots and must never lose their terminal outcomes to a timer firing after State shutdown. An ordinary handler error does *not* terminate the batch: the failing member closes with the existing error outcome (observed, as for any handler error) and the batch continues with its remaining members — only lease loss and coordination abandonment end a batch early, preserving the delivery guarantee. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, closing with the existing `OutcomePreempted`) and the batch terminates — remaining members that never ran close with a new closed-set terminal outcome, `skipped-lease-loss` (`OutcomeSkippedLeaseLoss`), never `OutcomePreempted` (they were not running) and never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. +A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. **Runtime Shutdown** must first close admission atomically — deliveries entering dispatch after shutdown begins are rejected at the gate, and outstanding preludes are synchronized before the drain starts, so no delivery can open a new window or tail after the drain has passed it — then seal any open window and drain it like every other detached tail: window timers are cancelled, and parked members close observably (the existing shutdown-drain semantics) before **Runtime State** cleanup. An open window's acknowledged members hold admission slots and must never lose their terminal outcomes to a timer firing after State shutdown. An ordinary handler error does *not* terminate the batch: the failing member closes with the existing error outcome (observed, as for any handler error) and the batch continues with its remaining members — only lease loss and coordination abandonment end a batch early, preserving the delivery guarantee. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, closing with the existing `OutcomePreempted`) and the batch terminates — remaining members that never ran close with a new closed-set terminal outcome, `skipped-lease-loss` (`OutcomeSkippedLeaseLoss`), never `OutcomePreempted` (they were not running) and never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end — and a completed member's event and closure references must be cleared from the batch's backing storage *before* its slot releases, so a released permit never has a still-reachable payload behind it (otherwise up to `MaxBurstBatch − 1` completed payloads per active batch would remain live beyond the cap) — except the batch's shared tail goroutine is itself covered: the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. From 5838bfeae734e8b8c5f61ac276c5ae729adf75ea Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 20:55:41 +0000 Subject: [PATCH 27/28] docs(adr): idle scope-coordinator removal; busy-post drain before adapter cleanup --- docs/adr/0015-runtime-coordination.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 3988655..4a1ce81 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -28,7 +28,7 @@ Semantics at the cap: **reject-with-signal, before ack and before dedupe marking This qualifies ADR 0003 explicitly (flagged per the domain docs' ADR-conflict rule): ADR 0003 defines a verified, normalized command as an **Accepted Event** owed acknowledgement, but the admission gate precedes acceptance — a delivery rejected at admission never becomes an **Accepted Event**, is never marked, and owes only the shape-aware overload response above. Overload rejection is a pre-acceptance outcome, not a broken acknowledgement of an accepted command. -For interaction shapes whose platform contract separates the acknowledgement from the user-visible response (ADR 0004: Slack `block_actions` requires an empty 2xx, with messages via `response_url`), the busy response cannot ride the acknowledgement body. The adapter acknowledges promptly per ADR 0004 and sends the busy message through the follow-up channel as a single, short-deadline, bounded adapter call under a small fixed busy-response budget — not a detached tail, so it does not consume `MaxDetached` capacity, and the budget caps its goroutines under saturation. The call runs on its own short-deadline context detached from the request context (which the platform cancels when the acknowledgement returns) and bounded by **Runtime Shutdown** — never on `r.Context()`, which would cancel the post, and never synchronously before the acknowledgement, which would delay it. If that budget is exhausted — or the interaction carries no follow-up channel at all (ADR 0004 treats `response_url` as optional) — the busy post is skipped: the rejection is still observed (`admission_rejected`), degrading only the user-visible signal, never the acknowledgement. These are the two narrow, honestly-stated exceptions to the visible-busy-response rule. +For interaction shapes whose platform contract separates the acknowledgement from the user-visible response (ADR 0004: Slack `block_actions` requires an empty 2xx, with messages via `response_url`), the busy response cannot ride the acknowledgement body. The adapter acknowledges promptly per ADR 0004 and sends the busy message through the follow-up channel as a single, short-deadline, bounded adapter call under a small fixed busy-response budget — not a detached tail, so it does not consume `MaxDetached` capacity, and the budget caps its goroutines under saturation. The call runs on its own short-deadline context detached from the request context (which the platform cancels when the acknowledgement returns) and bounded by **Runtime Shutdown** — never on `r.Context()`, which would cancel the post, and never synchronously before the acknowledgement, which would delay it. In-flight busy posts are tracked and drained before adapter cleanup under the shutdown deadline (their budget stays separate from `MaxDetached`), so `Shutdown` cannot return while a follow-up still uses adapter resources. If that budget is exhausted — or the interaction carries no follow-up channel at all (ADR 0004 treats `response_url` as optional) — the busy post is skipped: the rejection is still observed (`admission_rejected`), degrading only the user-visible signal, never the acknowledgement. These are the two narrow, honestly-stated exceptions to the visible-busy-response rule. Placement and validation: @@ -52,7 +52,7 @@ Interaction with each strategy (under `DispatchDeferred`): Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail) — and, identically, of ADR 0004's requirement that deferred interactions acquire the **Lock Lease** before acknowledgement, which likewise assumed the immediate-dispatch strategies — both flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: - **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). -- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): the incoming event is appended first, and a window that thereby reaches exactly `MaxBurstBatch` members **seals and rolls** — the sealed batch (cap-reaching member included) proceeds to dispatch, and the *next* event opens a new window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. The timer is **fixed, anchored at the window's first member**: later arrivals join the window but never reset it — resetting is debounce's semantic and would park the first accepted member indefinitely under steady sub-cap traffic — so every batch dispatches within one `DebounceInterval` of its first member unless the size cap seals it sooner. +- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): the incoming event is appended first, and a window that thereby reaches exactly `MaxBurstBatch` members **seals and rolls** — the sealed batch (cap-reaching member included) proceeds to dispatch, and the *next* event opens a new window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. A scope's coordination bookkeeping (window, timer, FIFO) is removed once the scope has no open window and no queued or running batch — synchronized against concurrent arrivals — so high-cardinality scope churn cannot grow idle coordinator state that `MaxDetached` does not count. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. The timer is **fixed, anchored at the window's first member**: later arrivals join the window but never reset it — resetting is debounce's semantic and would park the first accepted member indefinitely under steady sub-cap traffic — so every batch dispatches within one `DebounceInterval` of its first member unless the size cap seals it sooner. A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. **Runtime Shutdown** must first close admission atomically — deliveries entering dispatch after shutdown begins are rejected at the gate, and outstanding preludes are synchronized before the drain starts, so no delivery can open a new window or tail after the drain has passed it — then seal any open window and drain it like every other detached tail: window timers are cancelled, and parked members close observably (the existing shutdown-drain semantics) before **Runtime State** cleanup. An open window's acknowledged members hold admission slots and must never lose their terminal outcomes to a timer firing after State shutdown. An ordinary handler error does *not* terminate the batch: the failing member closes with the existing error outcome (observed, as for any handler error) and the batch continues with its remaining members — only lease loss and coordination abandonment end a batch early, preserving the delivery guarantee. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, closing with the existing `OutcomePreempted`) and the batch terminates — remaining members that never ran close with a new closed-set terminal outcome, `skipped-lease-loss` (`OutcomeSkippedLeaseLoss`), never `OutcomePreempted` (they were not running) and never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. From aef222e6fbc61754a4e224bbf690d9668f004f95 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 27 Aug 2026 21:05:02 +0000 Subject: [PATCH 28/28] =?UTF-8?q?docs(adr):=20decision-only=20re-scope=20p?= =?UTF-8?q?er=20maintainer=20ruling=20=E2=80=94=20decisions/invariants/non?= =?UTF-8?q?-goals;=20all=20mechanism=20prose=20cut;=20burst=20admission=20?= =?UTF-8?q?deferred=20to=20#53=20revival?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/adr/0015-runtime-coordination.md | 71 +++++++++++++-------------- 1 file changed, 34 insertions(+), 37 deletions(-) diff --git a/docs/adr/0015-runtime-coordination.md b/docs/adr/0015-runtime-coordination.md index 4a1ce81..a9c4c97 100644 --- a/docs/adr/0015-runtime-coordination.md +++ b/docs/adr/0015-runtime-coordination.md @@ -4,6 +4,8 @@ Proposed. This decides the deferred-dispatch admission bound (issue #44) and explicitly rejects, for now, the cross-instance coalescing State extension (issue #50) after a full design attempt — see the rejection section for the evidence and the reopening bar. Issue #50 stays open as gated future work. This ADR also gives the staged burst/preemption branch (PR #53) its verdict. +This is a decision-level document: it fixes decisions, invariants, and non-goals. Implementation mechanics — slot bookkeeping, timer and shutdown lifecycles, counter management — are deliberately not specified here; they are decided in the implementing PRs, where code and hardening tests can actually verify them against the invariants below. + ## Context ADR 0002 made `DispatchDeferred` the opt-in **Ack-Then-Work** mode: the adapter acknowledges after the synchronous prelude and the handler runs on the **Detached Work Context**, bounded by `DetachTimeout`. ADR 0012 expanded the **Concurrency Strategy** set; the reduced implementation on main ships `drop`, `queue`, `debounce`, and `concurrent` plus **Lock Scope** and universal lease-loss cancellation, with `burst` and force/steerability staged on PR #53 pending this design track. @@ -19,44 +21,36 @@ Both were drafted here as one design. The admission bound converged under review ### 1. Admission Bound (issue #44) -Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the slot-release point defined next (not until a terminal outcome is *recorded*; recording precedes cleanup). Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot **when the prelude returns** — the runtime's own boundary, before the adapter writes any response; an errored prelude is not acknowledged at all, and the runtime never observes the adapter's write, so prelude return is the only release point that cannot leak permits. Only routed deferred work holds its slot, and it holds it until the detached tail goroutine actually returns — after handler completion *and* lock cleanup — not merely until a terminal outcome is recorded: cleanup can stall (today's release path runs under an uncancellable context after the outcome is recorded), and a slot released before the goroutine, event, and closure are actually gone would let sustained traffic exceed the cap by the amount of work stuck in cleanup. - -Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to a shape-aware response (adapter-owned). Because the delivery was never acknowledged and never marked, no dedupe record blocks a retry — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away). The mapping is shape-aware because platforms do not retry every webhook shape: - -- **Platform-retried deliveries** (e.g. Slack Events API callbacks): the adapter returns the retry-inducing response (HTTP 429/503 equivalents) and the platform's own redelivery covers the event. -- **Direct user invocations that the platform does not redeliver** (e.g. Slack slash commands and interactivity — the in-repo Slack normalization records that these carry no retry headers): a bare 429 would turn a click into a silent permanent failure, so the adapter must answer with a *truthful busy response* to the user (a visible "busy, try again" acknowledgement, observed as rejected) — never a silent failure, never a fake success. The user's own retry is honored because no dedupe record exists. - -This qualifies ADR 0003 explicitly (flagged per the domain docs' ADR-conflict rule): ADR 0003 defines a verified, normalized command as an **Accepted Event** owed acknowledgement, but the admission gate precedes acceptance — a delivery rejected at admission never becomes an **Accepted Event**, is never marked, and owes only the shape-aware overload response above. Overload rejection is a pre-acceptance outcome, not a broken acknowledgement of an accepted command. +Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. It must be positive under `DispatchDeferred` (constructor validation, the `DetachTimeout` precedent); `DefaultRuntimeOptions` gains a default (1024). Optionally, `MaxDetachedPerTenant int` (0 = disabled) additionally caps any single installation's share, keyed on ADR 0006's installation identity `(adapter, tenant)` — a ceiling through the same rejection path, not a reservation. -For interaction shapes whose platform contract separates the acknowledgement from the user-visible response (ADR 0004: Slack `block_actions` requires an empty 2xx, with messages via `response_url`), the busy response cannot ride the acknowledgement body. The adapter acknowledges promptly per ADR 0004 and sends the busy message through the follow-up channel as a single, short-deadline, bounded adapter call under a small fixed busy-response budget — not a detached tail, so it does not consume `MaxDetached` capacity, and the budget caps its goroutines under saturation. The call runs on its own short-deadline context detached from the request context (which the platform cancels when the acknowledgement returns) and bounded by **Runtime Shutdown** — never on `r.Context()`, which would cancel the post, and never synchronously before the acknowledgement, which would delay it. In-flight busy posts are tracked and drained before adapter cleanup under the shutdown deadline (their budget stays separate from `MaxDetached`), so `Shutdown` cannot return while a follow-up still uses adapter resources. If that budget is exhausted — or the interaction carries no follow-up channel at all (ADR 0004 treats `response_url` as optional) — the busy post is skipped: the rejection is still observed (`admission_rejected`), degrading only the user-visible signal, never the acknowledgement. These are the two narrow, honestly-stated exceptions to the visible-busy-response rule. +Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`); the webhook layer maps it to a shape-aware response (adapter-owned): -Placement and validation: +- **Platform-retried deliveries** (e.g. Slack Events API callbacks): a retry-inducing response (HTTP 429/503 equivalents); the platform's own redelivery covers the event, and because the delivery was never marked, that retry is not deduped away — the same honesty contract prelude errors already follow. +- **Direct user invocations the platform does not redeliver** (e.g. Slack slash commands and interactivity, which carry no retry headers): a truthful, user-visible busy signal, delivered per each shape's acknowledgement contract on a best-effort, bounded basis — never a silent failure, never a fake success. Where the platform's contract makes a visible signal impossible, the rejection is still observable. -- The gate sits at the head of the prelude, before any **Runtime State** interaction, so a rejected delivery leaves no record. A consequence, stated as an explicit qualification of ADR 0002's duplicate contract (per the domain docs' ADR-conflict rule): the required `State` contract has no read-only **Event Identity** check (`MarkEvent` writes on a miss), so at the cap the gate cannot distinguish a redelivery of an already-marked event — under saturation, duplicates receive the overload response like any other delivery and converge to the ordinary duplicate acknowledgement once capacity frees. Probing dedupe with `MarkEvent` is prohibited: marking a first delivery before rejecting it would let the platform's retry be acknowledged as a duplicate and never handled — silent event loss, strictly worse than extra retries. If saturation retry amplification proves material in practice, a read-only dedupe check can be added later as a small optional State capability; it is deliberately not part of this ADR. -- `MaxDetached` must be positive under `DispatchDeferred` (constructor validation, the `DetachTimeout` precedent); `DefaultRuntimeOptions` gains a default (1024). `DispatchSync` ignores it: a synchronous delivery's goroutine and payload belong to the HTTP request before dispatch begins, so a runtime-side cap cannot shed them — and `net/http` imposes no request-concurrency limit by itself. Bounding synchronous serving is therefore an explicit operator contract at the HTTP layer (a request/connection limiter in front of the **Webhook Handler**), stated in the option's GoDoc rather than assumed — see Non-goals. -- Observability: a rejected delivery emits a new observation (`admission_rejected`, carrying the adapter and tenant attributes) and closes its dispatch span with a new terminal outcome (`admission-rejected`). Rejections are never silent. -- Multi-tenant fairness: `MaxDetached` alone is a global ceiling, so on a multi-tenant runtime (ADR 0006) one tenant's sustained valid traffic could starve co-located tenants. An optional `MaxDetachedPerTenant int` (**Runtime Options**, 0 = disabled, negative rejected at **Runtime Construction**) bounds any single installation's share of the admitted work using the same rejection path and observability. Accounting keys on the composite `(adapter, tenant)` — ADR 0006's installation identity — never the bare tenant string, so same-named tenants on different adapters do not share a bucket; an empty tenant is counted as that adapter's single untenanted bucket (it must not bypass the limit), meaning per-installation isolation is only meaningful for adapters that populate `Event.Tenant`. A counter entry is removed when its installation's last slot releases — tenant-churn traffic must not grow accounting state without bound. It is a sublimit, not a reservation — fair-share scheduling is out of scope. Deployments serving a single tenant need only the global bound. +This qualifies two accepted ADRs, flagged per the domain docs' ADR-conflict rule: **ADR 0003** — the admission gate precedes acceptance, so a rejected delivery never becomes an **Accepted Event** and owes only the overload response, and **ADR 0002** — the required `State` contract has no read-only dedupe check, so under saturation a redelivered duplicate receives the overload response like any other delivery and converges to the ordinary duplicate acknowledgement once capacity frees. Probing dedupe with `MarkEvent` is prohibited: marking a first delivery before rejecting it would let the platform's retry be deduped away unhandled — silent event loss, strictly worse than extra retries. -Interaction with each strategy (under `DispatchDeferred`): +**Invariants** — binding on every implementing PR, verified there by code and hardening tests: -| Strategy | What retains memory | What the bound caps | -|---|---|---| -| drop | running tails (one per scope) | total tails across scopes | -| queue | running tails + at most one parked waiter per scope | scope-cardinality floods | -| debounce | parked waiters (≤ 1 per scope) + running tails | scope-cardinality floods | -| concurrent | slot-waiters + at most `MaxConcurrent` running | the waiting line behind `MaxConcurrent` | -| burst (staged) | batch members (payload retained, no goroutine) + the running member | total retained members on the instance | +1. **Bounded retention.** Everything an admitted deferred delivery retains (goroutine, payload, closure — running tails, parked waiters, slot-waiters, future batch members) counts against the cap, and capacity is never released while that work remains retained. +2. **Honest rejection.** A rejected delivery is never acknowledged as handled, never marked in **Event Identity** dedupe, and always observable: a new observation (`admission_rejected`, carrying adapter and tenant) and a new terminal dispatch outcome (`admission-rejected`). +3. **No silent loss.** Every admitted delivery reaches an observable terminal outcome — including under **Runtime Shutdown**. -### 2. Burst shaping (issue #44's scope extension) +Per-strategy invariant under `DispatchDeferred` (one line each): -Burst requires `DispatchDeferred` (constructor validation, the debounce precedent): a synchronous webhook cannot park batch members past the platform acknowledgement deadline, and the burst invariants below are defined only for deferred dispatch. Lock sequencing follows the shipped debounce precedent exactly: burst takes no **Thread Lock** in the prelude — members ack promptly and park; the batch tail acquires the lock once, before running its members. This extends to burst the same qualification of ADR 0002's pre-ack-acquisition rule that the merged debounce implementation already established (main's prelude acquires the lock only under drop/queue; debounce coordinates in the tail) — and, identically, of ADR 0004's requirement that deferred interactions acquire the **Lock Lease** before acknowledgement, which likewise assumed the immediate-dispatch strategies — both flagged per the domain docs' ADR-conflict rule: ADR 0002's rule governs the immediate-dispatch strategies, and the coalescing strategies acquire at dispatch time in the tail — still exactly one lock owner per scope, still lease-loss-cancelled. When burst ships (see the PR #53 outcome), batch growth is bounded twice: +| Strategy | What the bound caps | +|---|---| +| drop | total running tails across scopes | +| queue | running tails plus parked waiters (≤ 1 per scope) under scope-cardinality floods | +| debounce | parked waiters (≤ 1 per scope) plus running tails under scope-cardinality floods | +| concurrent | the waiting line behind `MaxConcurrent`, plus the running set | +| burst (staged) | all retained batch members — decided at #53 revival under the invariants above | -- **Globally** by `MaxDetached`: each member counts from admission until its member-run ends (the final member until the batch tail goroutine returns — below). -- **Per scope** by a new `MaxBurstBatch int` (**Runtime Options**, required positive under burst): the incoming event is appended first, and a window that thereby reaches exactly `MaxBurstBatch` members **seals and rolls** — the sealed batch (cap-reaching member included) proceeds to dispatch, and the *next* event opens a new window. Nothing is dropped at the batch layer; overflow only ever moves batch boundaries. (#44 demanded an explicit overflow policy; seal-and-roll is it.) Sealed batches for a scope dispatch in seal order through a per-instance FIFO: a rolled batch's tail runs only after its predecessor's tail returns. A scope's coordination bookkeeping (window, timer, FIFO) is removed once the scope has no open window and no queued or running batch — synchronized against concurrent arrivals — so high-cardinality scope churn cannot grow idle coordinator state that `MaxDetached` does not count. Batches never enter the pending-waiter supersession path — supersession is newest-wins and would drop accepted members, violating the delivery-preserving guarantee — and FIFO ordering prevents a newer batch from racing an older one to the **Thread Lock**. The window interval is `DebounceInterval`, reused under burst as ADR 0012's `debounceMs` prescribes ("waits `debounceMs` on an idle scope") and required positive under burst like under debounce; its GoDoc's "ignored otherwise" note updates accordingly. The timer is **fixed, anchored at the window's first member**: later arrivals join the window but never reset it — resetting is debounce's semantic and would park the first accepted member indefinitely under steady sub-cap traffic — so every batch dispatches within one `DebounceInterval` of its first member unless the size cap seals it sooner. +`DispatchSync` is out of scope by decision: a synchronous delivery's goroutine and payload belong to the HTTP request before dispatch begins, so a runtime cap cannot shed them — bounding synchronous serving is a serving-layer operator contract (see Non-goals). -A member's *execution* is budgeted per member, but its *retention* is batch-derived, and this explicitly qualifies ADR 0002's per-tail lifetime rule for burst (flagged per the ADR-conflict rule): a late member of a slow batch waits through its predecessors, so a burst member's admission-to-return lifetime is bounded by the batch bound — the window interval, plus a coordination budget, plus up to `MaxBurstBatch` sequential member budgets (plus any predecessor batches in the FIFO, each themselves so bounded) — not by one `DetachTimeout`. The coordination budget makes the bound real: a batch's pre-execution wait (FIFO turn plus **Thread Lock** acquisition) is bounded by one `DetachTimeout`, and a batch that cannot acquire within it is **abandoned observably** — every member closes with the existing abandonment outcome, slots released at tail return — matching the shipped single-waiter abandonment semantics rather than parking forever behind a stuck scope. **Runtime Shutdown** must first close admission atomically — deliveries entering dispatch after shutdown begins are rejected at the gate, and outstanding preludes are synchronized before the drain starts, so no delivery can open a new window or tail after the drain has passed it — then seal any open window and drain it like every other detached tail: window timers are cancelled, and parked members close observably (the existing shutdown-drain semantics) before **Runtime State** cleanup. An open window's acknowledged members hold admission slots and must never lose their terminal outcomes to a timer firing after State shutdown. An ordinary handler error does *not* terminate the batch: the failing member closes with the existing error outcome (observed, as for any handler error) and the batch continues with its remaining members — only lease loss and coordination abandonment end a batch early, preserving the delivery guarantee. Mid-batch lease loss has one disposition: the running member is cancelled (`ErrPreempted`, closing with the existing `OutcomePreempted`) and the batch terminates — remaining members that never ran close with a new closed-set terminal outcome, `skipped-lease-loss` (`OutcomeSkippedLeaseLoss`), never `OutcomePreempted` (they were not running) and never silently discarded; there is no reacquisition, preserving the single-acquisition lifecycle. The bound is cooperative, like every runtime stop: `DetachTimeout` expires a member's context but Go cannot force a handler to return, so a cancellation-ignoring member retains itself and every later member indefinitely — that, plus the §1 stalled-cleanup case, are the two unbounded residuals, and operators sizing capacity from the formula assume handlers honor cancellation. Operators size `MaxBurstBatch` and `DetachTimeout` together with the product in mind; ADR 0002's single-`DetachTimeout` lifetime keeps governing every non-burst tail. +### 2. Burst admission semantics: deferred to the PR #53 revival -Two rules carried from the #38 review history: each batch member runs with its own fresh `DetachTimeout` budget — a batch never runs under a shared deadline inherited from its first member ([r3871403320](https://github.com/coder/chat/pull/38#discussion_r3871403320)) — and the batch's lock lifecycle reuses the same refresh/cancel/outcome machinery as every other deferred holder ([r3872272204](https://github.com/coder/chat/pull/38#discussion_r3872272204)): one lease-lifecycle implementation, no parallel loop. Slot accounting follows §1's release rule at the batch level too: members release as their member-runs end — and a completed member's event and closure references must be cleared from the batch's backing storage *before* its slot releases, so a released permit never has a still-reachable payload behind it (otherwise up to `MaxBurstBatch − 1` completed payloads per active batch would remain live beyond the cap) — except the batch's shared tail goroutine is itself covered: the final member's slot is held until that goroutine actually returns after lock cleanup, so a stalled batch cleanup still counts against the cap. Byte-based accounting is refused — see Non-goals. +Burst's admission interaction — window and batch caps, member accounting, batch lifecycle, lock sequencing, and any ADR 0002/0004 qualification it requires — is decided when burst revives (see the PR #53 outcome), under this ADR's invariants plus one burst-specific invariant fixed now: **batch shaping is delivery-preserving** — any overflow policy may move batch boundaries but never drops an accepted member. Nothing else about burst is specified here. ### 3. Per-instance supersession is the v0.x contract @@ -84,18 +78,17 @@ A full State-backed design was drafted and reviewed on this very PR: per-scope f | Class | Disposition in this design | |---|---| -| Unbounded admission | Precluded: pre-ack `MaxDetached` gate; per-scope `MaxBurstBatch` seal-and-roll | +| Unbounded admission | Precluded: pre-ack `MaxDetached` gate; the bounded-retention invariant; delivery-preserving batch shaping decided at #53 revival | | Process-local coordination with distributed semantics implied | Precluded by scope honesty: v0.x claims per-instance semantics only (§3); no distributed protocol ships without the formal bar | | Key-only force release | Rejected finally; no force primitive ships in v0.x | | Non-atomic ownership handoff; premature handover | Not shipped: no preemption, no local preemption registries in v0.x | -| Lease-lifecycle divergence | Precluded by rule: all lock-holding paths (including future burst) reuse the one refresh/cancel/outcome implementation | -| Temporal/budget skew | Precluded where in scope: per-member execution budgets, no shared batch deadline; local admission-sequence ordering already on main | +| Lease-lifecycle divergence; temporal/budget skew | Deferred with the burst revival, bound by this ADR's invariants and decided against code and hardening tests | ## Outcome for PR #53 (staged burst + preemption) **Verdict: close PR #53.** Judged against per-instance semantics plus the admission bound: -- **Burst: revive with changes, as its own PR, after the admission bound lands.** Burst is per-instance batching and needs no cross-instance machinery. Required changes: members count against `MaxDetached`; `MaxBurstBatch` with seal-and-roll; per-member fresh `DetachTimeout` budgets; the batch lock lifecycle must reuse the shared refresh/cancel machinery (the staged branch's separate refresh loop reproduced the lease-lifecycle failure class twice); `DispatchDeferred` required at construction. +- **Burst: revive with changes, as its own PR, after the admission bound lands.** Burst is per-instance batching and needs no cross-instance machinery. The revival decides burst's admission semantics and lifecycle under this ADR's invariants (bounded retention, honest rejection, no silent loss, delivery-preserving shaping), with code and hardening tests as the verification medium — not ADR prose. - **Preemption: rejected pending the formal-design bar.** Key-only `ForceReleaseLock` and the local preemption choreography are rejected finally (above); a safe replacement is a distributed protocol of exactly the class this ADR declines to specify in prose. `ErrPreempted`/`OutcomePreempted` (already on main, serving lease-loss cancellation) are unaffected. ## Non-goals @@ -106,18 +99,18 @@ This design explicitly refuses to promise: - **Admission control for synchronous dispatch.** Under `DispatchSync` the goroutine and payload exist at the HTTP layer before the runtime sees the delivery; a runtime cap cannot shed that load, and `net/http` imposes no request-concurrency limit of its own. Bounding synchronous serving is an explicit serving-layer operator contract. - **Byte-accounted admission.** The runtime cannot meaningfully measure retained `Raw` platform payloads plus handler closures; the bound is a count, and operators size it against their platform's payload ceiling. - **Fleet-wide admission control.** `MaxDetached` protects one instance's memory; fleet capacity management belongs to the operator's front door. -- **Fair-share tenant scheduling.** `MaxDetachedPerTenant` is a per-tenant ceiling, not a reservation or weighted scheduler; guaranteed tenant throughput under saturation is not promised. +- **Fair-share tenant scheduling.** `MaxDetachedPerTenant` is a per-installation ceiling, not a reservation or weighted scheduler; guaranteed tenant throughput under saturation is not promised. - **Exactly-once dispatch.** ADR 0002's crash-loss contract for deferred dispatch is inherited unchanged. ## Consequences -- Three new **Runtime Options** (`MaxDetached`, optional `MaxDetachedPerTenant`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission (plus, when burst ships, the `skipped-lease-loss` terminal outcome). `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. -- Adapters gain one honesty duty: map `ErrAdmissionRejected` shape-awarely — a retry-inducing response for platform-redelivered shapes, a truthful busy response for direct invocations the platform does not redeliver. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. +- New **Runtime Options** (`MaxDetached`, optional `MaxDetachedPerTenant`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR. +- Adapters gain one honesty duty: map `ErrAdmissionRejected` shape-awarely per §1. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc. - The `State` interface does not change, and no optional State capability ships with this ADR. - Multi-instance deployments keep the documented per-instance coalescing semantics indefinitely, until the reopening bar is met. Operators who need stronger ordering today must route same-scope traffic to one instance (sticky routing) — an operational workaround, stated honestly, not a runtime promise. -- Three ADR 0012 statements are superseded here (flagged per the domain docs' ADR-conflict rule): its proposed force surface (`ForceReleaseLock` by key) falls to the final rejection above; its staging gate that held `burst` on "fenced-coordination design work" is lifted — burst ships per-instance under §2, gated only on the admission bound; and its expectation that `queue`/`debounce`/`burst` "need wait/coalesce coordination" expanding every State implementation is withdrawn for v0.x — per-instance supersession (§3) needs no State expansion, and any future coordination contract goes through the reopening bar. ADR 0003 is qualified as described in §1. +- Three ADR 0012 statements are superseded here (flagged per the domain docs' ADR-conflict rule): its proposed force surface (`ForceReleaseLock` by key) falls to the final rejection above; its staging gate that held `burst` on "fenced-coordination design work" is lifted — burst is gated only on the admission bound and its own revival decisions; and its expectation that `queue`/`debounce`/`burst` "need wait/coalesce coordination" expanding every State implementation is withdrawn for v0.x — per-instance supersession (§3) needs no State expansion, and any future coordination contract goes through the reopening bar. ADR 0003 and ADR 0002 are qualified as described in §1. - The CONTEXT.md glossary gains **Admission Bound** when this ADR is accepted and implementation lands. -- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) burst revival per the PR #53 outcome. Issue #50 remains open, labeled as gated on the reopening bar. +- Implementation sequencing, each step its own gated PR: (1) admission bound — closes #44; (2) burst revival per the PR #53 outcome, deciding burst admission semantics under this ADR's invariants. Issue #50 remains open, labeled as gated on the reopening bar. ## Alternatives Considered @@ -133,6 +126,10 @@ Rejected. Parking the webhook goroutine busts platform ack deadlines (Slack's 3s Rejected as the whole answer. A load balancer cannot see detached-tail occupancy — the exhausted resource is invisible outside the runtime. Front-door rate limiting remains complementary. +### Specify the full admission/burst mechanism in this ADR + +Rejected, twice over, by this PR's own review history: first the cross-instance protocol and then the burst lifecycle accumulated new findings with every round of prose specification. Mechanism belongs where it can be verified — code and hardening tests in the implementing PRs — while this document fixes only the decisions and invariants those PRs must satisfy. + ### Ship the cross-instance coalescing protocol now (the original scope of this ADR) Rejected — this is the decision the rejection section records. The drafted protocol (fences, registration TTLs, fenced takeover, reconciliation) accumulated new P1-severity holes in every prose review round without converging; correctness of this protocol class must be established by model checking or a proven primitive, not by iterative prose review.