Repository navigation
fix(waits): enforce wait deadlines, fail empty races and memoize child timeouts - #99
Merged
Merged
Conversation
psteinroe
added a commit
that referenced
this pull request
Oct 20, 2026
…nalWithStart and ctx.emit (#100) Adds a `{ signal: name }` branch to `ctx.waitForAny()`. A keyed session started with `signalWithStart` can now suspend until its next input or its tool execution, whichever comes first, instead of ending: ```ts ctx.onSignal("input", (payload, { id }) => saveInput(id, payload)); const winner = await ctx.waitForAny(`idle:${n}`, { input: { signal: "input" }, // new tool: { execution: toolExecutionId }, }); // { key: "input", signal: { id, name } } | { key: "tool", outcome } | { key: "timeout" } ``` **Registration and wake (SQL)** A signal branch is an `execution_wait` subscription with a new `signal_name` column. Its step key is `<stepKey>:<branch>`, the same as execution branches. `_private_register_event_race` holds the claimed execution's row lock. While holding it, the race wins at once if the execution has an undelivered signal with that name; otherwise it subscribes. Senders (`signal_execution`, `signal_with_start`, so also `conductor.signal`, `ctx.sendSignal` and `ctx.signalWithStart`) already lock that row and bump `signal_count`. Under that same lock, `_private_store_signal` now does three things for a new signal: it stores `{status: 'resolved', signal: {id, name}}` under the step key of every unexpired matching subscription, removes those subscriptions, and wakes the execution if it is suspended at a wait (the same condition dispatch uses). Every signal is ordered against registration by the row lock, so a signal stored between the check and the suspend wakes the execution. No new lock is taken, and the sender still locks only the signalled row, which is documented in the AGENTS.md lock-order section. Signals for names no waiting branch listens for behave as before: they are kept for the next attempt and do not wake the execution. Branches are registered only on the first call, the same as execution branches in #99. A signal that arrives after the deadline does not win (`expires_at >= now`, matching #99's rule for result waits). Winner selection, the timeout, closing losing branches and removing their subscriptions all reuse the existing race code, so the one-winner guarantee from #79 holds. **The branch does not consume the signal** The signal stays undelivered. The next attempt primes it at claim and delivers it to `ctx.onSignal()`. The result carries only the signal's `{ id, name }`. **Deviation from the brief: signals being handled count too** "Undelivered" alone leaves a lost wake-up. A handler can return and acknowledge a signal after the user's loop checked its saved inputs but before the race registers. The wait then suspends with an unprocessed input and nothing to wake it. To close that gap, `SignalInbox.hold()` stops new handlers from starting between the `waitForAny()` call and the end of registration. The signal whose handler is still running at the call is passed as `p_delivering_signal_id` and counts like an undelivered one. The documented rule is: a signal wins unless its handler returned before the call. Docs also say to register a handler for every signal branch. Without one, the signal is never delivered, so it wins every later wait at once. **Ride-along: `metadata` for `ctx.signalWithStart` and `ctx.emit`** Both now take `metadata?: Metadata | ((metadata) => Metadata)`, the same option as `ctx.invoke` and `ctx.invokeAndWait`. It is resolved by the same `childMetadata()` helper, so it is validated against the Conductor's `metadata` schema and the 8 KB limit, and it defaults to this execution's metadata. `ctx.signalWithStart` applies it only when the call starts the execution, and the started id stays memoized under the step. A running execution keeps its metadata. For `ctx.emit` it becomes the event's metadata and flows to the executions the event triggers. `ctx.sendSignal` starts nothing, so it gets no option, and the docs say so. `tests/integration/execution-metadata.test.ts` covers an override, the function form, inheritance by default, a signal to a running execution that keeps its metadata, and schema-validation failures for both. `execution-metadata-types.test.ts` covers the types. **Verification** New `tests/integration/wait-for-signal.test.ts`: - a signal stored before the wait wins without suspending (handler entered once) - a signal whose handler is running at the call wins, and so does one whose handler returns and is acknowledged while the step read is held back. The second test fails if the hold starts after `loadStep`, and the first fails without the delivering id. - a signal wakes a suspended wait; replays return the memoized winner; `onSignal` still receives the signal - two deterministic races on separate connections with held transactions: a signal blocked behind an uncommitted registration wakes it, and a registration blocked behind an uncommitted signal wins at once - one winner when a signal and an execution branch settle at the same moment (two orchestrators) - `signalWithStart` wakes a keyed session waiting on input or its tool - a signal for another name does not wake the wait (`run_at` stays `infinity`) - under a set test clock: a signal wakes the wait. After advancing past the timeout, a late signal does not win and the timeout does. This test fails without the expiry check. Unit tests cover `SignalInbox.hold()` and the `WaitForAnyResult` type for signal branches. `just lint`, `just format`, `pnpm typecheck` and the full `pnpm test` (579 tests, plus the new ones) pass. **Follow-ups / interaction with open PRs** - #98 (`steps.seq`): when it lands, both new step inserts (in `_private_store_signal` and the race's immediate-signal path) need `seq` from `_private_event_seq`. Without it, a signal and an execution resolved at the same frozen time fall back to branch order. I'll add that on rebase if #98 lands first. - #99: compatible. Signal branches already follow its first-call-only registration and deadline rules.
…d timeouts - result waits store no outcome after their deadline (C11) - waitForAny without an open branch raises instead of suspending forever (C12) - a caught invokeAndWait timeout is memoized and no longer fails later child calls (C23) - conductor.waitForResult cancels a blocked read on timeout (O4-08) - durations take DurationInput everywhere; timeout: 0 means zero (D5c)
psteinroe
force-pushed
the
fix/wait-deadlines
branch
from
October 20, 2026 19:14
58402c2 to
1552136
Compare
psteinroe
added a commit
that referenced
this pull request
Oct 20, 2026
… test clock (#98) ## What Under a set test clock (`createTestClock`, `db.client.setFakeTime`), event waits never received events emitted after them. Dispatch requires `source.created_at > subscription.created_at`, and `_private_current_time()` returned the frozen `fake_now` as-is, so the subscription and the later event got the same timestamp and the event was dropped. This affected `ctx.waitForEvent()`, `ctx.subscribe()` + `ctx.waitForAny()`, and anything else that waits on events while the clock is set. `waitForAny` also picked the wrong winner: it orders resolved branches by step `created_at`, so ties fell back to branch order. ## Fix: fake time keeps running Production code and schema are unchanged; only fake time changes. - `_private_current_time()` returns `clock_timestamp() + pgconductor.fake_offset` when the setting is set. It replaces the `pgconductor.fake_now` setting. - `createTestClock`: `set(date)` stores `date - clock_timestamp()` as the offset, `advance(d)` adds `d` to it, and `reset()` clears it. `now()` applies the same offset to `Date.now()`, rounded up to whole milliseconds so it never reads before the time set, so database and worker time run on together (the worker's sampled offset stays ~0). The offset is computed once in a materialized CTE. Without that, Postgres inlines the subquery and `advance` reads the setting it just wrote. - `db.client.setFakeTime()` / `clearFakeTime()` use the same setting. I first pushed a fix that added a sequence (`_private_event_seq`, with seq columns on subscriptions and steps, and in the dispatch payload). I dropped it: it changed the production schema for a problem that only exists in tests. ## Behaviour change (test clock) **Fake time no longer stands still.** After `clock.set(date)` or `advance()`, time keeps running from that point like real time. Tests that compare exact timestamps after setting the clock need bounds instead (`toBeGreaterThanOrEqual`). I updated the docs (`api/testing.md`, `crafting-tasks/testing.md`) and AGENTS.md. Tests updated for this change: - `testing.test.ts`: exact `completedAt` and `current_setting('pgconductor.fake_now')` checks now use bounds; added "fake time keeps running from the time it was set to" - `orchestrator-cleanup.test.ts`, `window-execution.test.ts`, `dead-letter.test.ts`: exact timestamp checks now use bounds - `batch-processing.test.ts`: the batch sleep test advanced exactly 1s past a 1s sleep; it now uses 2s steps - `wait-deadlines.test.ts` (from #99): the exact `ctx.sleep` deadline check now uses bounds ## Verification - New `wait-for-event` tests, each run with fake time and with real time: - `waitForEvent` receives an event emitted after it - `subscribe` + `waitForAny` receive an event emitted after subscribe - `waitForAny` prefers the branch stored first - an event emitted before the wait, and held back so it is dispatched only after the subscription exists, is ignored - The fake-time versions failed before the fix (the event was never delivered); the real-time versions passed. - `just lint`, `just format`, `pnpm typecheck`, `pnpm test` (612 tests) pass. ## Follow-ups / notes - With real time, the "after the subscription" check still relies on `clock_timestamp()` being monotonic, so a wall-clock step (NTP) could misorder an event and a subscription. That was already the case before this PR. - The index `idx_custom_event_subscription_wait_match (event_key, created_at, expires_at, id)` does not seem to be used by any query (dispatch joins subscriptions by id). - `throttle-debounce` "debounce with partition keys" failed once in CI on Postgres 15. It uses real time (no fake clock), and two invokes can fall into different 60s debounce slots. It looks like an existing flake, unrelated to this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the wait deadline and timeout findings from the review: C11 (O3-02), C12 (O5-03), C23 (O5-04 plus the memoization gap R4 found), O4-08, and owner decision D5c (durations).
C11: result waits accepted outcomes that settled after their deadline
Root cause.
_private_resolve_result_waitsstored an outcome without looking at the wait'sexpires_at. On top of that, when the waiter resumed,_private_register_event_racewent over every branch again, stored the outcome of any execution that had settled by then, and picked a winner before it checked the timeout. So an execution that settled after the deadline, but before an unpolled waiter resumed, still won.Fix.
_private_resolve_result_waitsjoins each wait to its subscription and stores the outcome only ifexpires_at is null or expires_at >= now. Every path that ends an execution calls it at the moment the outcome settles, so the check compares the deadline with the outcome's terminal time, not with when the waiter resumes. The bound is inclusive, the same as dispatch uses for events (created_at <= expires_at)._private_register_event_raceregisters an execution branch only on its first call, meaning branches that have neither a stored step nor a subscription. A branch that already has a subscription gets its outcome from the path that ends the execution, which checks the deadline. On the first call, an execution that has already settled still resolves at once. The unknown-execution check also covers only new branches, so an awaited execution that settles late and is then removed by retention gives a timeout, not "not found".C12:
waitForAnywith no open branch ignored its timeoutRoot cause. A race keeps its deadline only on its branch subscriptions. A race with no branches, or only branches that an earlier race closed, has no subscription to hold the deadline. So every resume computed a new deadline and suspended again, forever, without using up attempts.
Decision: fail fast. If a race has no winner and no open subscription,
_private_register_event_raceraiseswaitForAny "<stepKey>" has no open branch. This is always a programming error. Honouring the timeout instead would need race state stored separately, only to always return{ key: "timeout" }. Documented in thectx.waitForAny()reference.C23: a caught
invokeAndWaittimeout broke later child calls and was not memoizedRoot cause.
invokeAndWaittreated "the claimed snapshot haswaiting_on_execution_id" as "this call timed out". So a secondinvokeAndWaitin the same attempt threw the same timeout without starting its child. The timeout itself was also never stored: onceclearWaitingStatehad cleared the waiting fields, a later replay invoked the timed-out child a second time and waited on it again.Fix. Following R4's note, clearing the snapshot alone is not enough; the timeout has to be stored.
invokeAndWaitnow stores{status: 'completed', result}(settlement) or{status: 'timed_out'}. The raw child result could not be told apart from a timeout marker.clearWaitingStatestores the timeout under the parent'swaiting_step_keyin the same statement that cancels the child and clears the wait. It is fenced by the claim token and returns whether it still held the claim. If not, the attempt hangs up as suspended instead of throwing.invokeAndWaittreats a missing result as a timeout only when the claim'swaiting_step_keyis its own key. A claim keeps that key only while the execution waits on a child. Any other key starts the child.O4-08:
conductor.waitForResulttimeout during a blocked readgetExecutionnow cancels its query (postgres.jsquery.cancel()) when the signal aborts.waitForResultrejects with the abort reason (aTimeoutErrorfor the timeout) whether the read fails because of the cancel or returns after the deadline. Cancelling on abort applies only to this read; settlement still runs under the worker's aborted signal, as O4-08 asked.D5c: durations
ctx.sleep(),BatchTaskContext.sleep(), thectx.invokeAndWait()timeout and theconductor.waitForResult()timeout now take aDurationInput(numbers are milliseconds, or a string such as"30s"), parsed withparseDuration(). The event and result waits already took one. Config fields whose name carries the unit (timeoutMs,pollIntervalMs,batch.timeoutMs,throttle.seconds,removeOnComplete.days) are unchanged.Behaviour changes
timeout: 0means zero, never infinity. Before,ctx.invokeAndWait(..., { timeout: 0 })waited forever, andconductor.waitForResult(id, { timeout: 0 })had no timeout. Now the first times out at once and cancels the child, and the second rejects with aTimeoutErrorat once. Omittimeoutto wait indefinitely.parseDuration). Before, they were passed to SQL orsetTimeout.invokeAndWaittimeout is thrown again on every replay, instead of invoking the child again.waitForAnywith no open branch throws instead of waiting.invokeAndWaitsteps change shape ({status, result}). The migration is edited in place, as the project does during active development. A database with child waits in flight from an older build would read their stored result wrongly.Tests
tests/integration/wait-deadlines.test.tsbrings in the R3/R4 reproductions with their barriers and fake time unchanged:mainand pass now.maxAttempts: 1.errorsholds one entry per run of the handler, and thesleepingflag (also true for a completed parent) is no longer asserted.access exclusivelock on the executions table.waitForResult(id, { timeout: 100 })must reject withTimeoutErrorwhile the lock is still held, and the connection is usable afterwards. Without the change the read stays blocked.invokeAndWaitwithtimeout: 0times out at once and cancels the child.conductor.waitForResultwithtimeout: 0rejects.ctx.sleep("nap", "1m")reschedules by one minute.The existing test for a first registration against an execution that has already settled still passes.
just lint,just format,pnpm typecheckand the fullpnpm testpass. One existing test (wait-for-result.test.ts, "a pending child cancelled when its parent's wait times out") setwaiting_on_execution_idby hand withoutwaiting_step_key, a state settlement never produces; it now sets both.Hot paths
parent_steps_allchanges (it wraps the result injsonb_build_object); the plan does not change._private_resolve_result_waitsruns only when a settled execution has waiters. Its new join is a lookup on the unique(execution_id, step_key)index._private_register_event_raceandclearWaitingStaterun once per wait, not per claim or settlement.Follow-ups (out of scope)
invokeAndWaitdeadlines in settlement. A child that completes after the parent's deadline, but before the parent resumes, still delivers its result (completed_parentsdoes not compareparent.run_atwith now). A child that fails late still fails the parent through the cascade. Applying the same terminal-time rule there is a change to settlement and deserves its own PR.conductor.waitForResultlogs the cancelled read at error level throughDatabaseClient.query()("canceling statement due to user request"). It is harmless, but noisy if callers often time out against a blocked database.