Skip to content

fix(waits): enforce wait deadlines, fail empty races and memoize child timeouts - #99

Merged
psteinroe merged 1 commit into
mainfrom
fix/wait-deadlines
Oct 20, 2026
Merged

psteinroe merged 1 commit into
mainfrom
fix/wait-deadlines

Conversation

@psteinroe

Copy link
Copy Markdown
Owner

Fixes the wait deadline and timeout findings from the review: C11 (O3-02), C12 (O5-03), C23 (O5-04 plus the memoization gap R4 found), O4-08, and owner decision D5c (durations).

C11: result waits accepted outcomes that settled after their deadline

Root cause. _private_resolve_result_waits stored an outcome without looking at the wait's expires_at. On top of that, when the waiter resumed, _private_register_event_race went over every branch again, stored the outcome of any execution that had settled by then, and picked a winner before it checked the timeout. So an execution that settled after the deadline, but before an unpolled waiter resumed, still won.

Fix.

  • _private_resolve_result_waits joins each wait to its subscription and stores the outcome only if expires_at is null or expires_at >= now. Every path that ends an execution calls it at the moment the outcome settles, so the check compares the deadline with the outcome's terminal time, not with when the waiter resumes. The bound is inclusive, the same as dispatch uses for events (created_at <= expires_at).
  • _private_register_event_race registers an execution branch only on its first call, meaning branches that have neither a stored step nor a subscription. A branch that already has a subscription gets its outcome from the path that ends the execution, which checks the deadline. On the first call, an execution that has already settled still resolves at once. The unknown-execution check also covers only new branches, so an awaited execution that settles late and is then removed by retention gives a timeout, not "not found".

C12: waitForAny with no open branch ignored its timeout

Root cause. A race keeps its deadline only on its branch subscriptions. A race with no branches, or only branches that an earlier race closed, has no subscription to hold the deadline. So every resume computed a new deadline and suspended again, forever, without using up attempts.

Decision: fail fast. If a race has no winner and no open subscription, _private_register_event_race raises waitForAny "<stepKey>" has no open branch. This is always a programming error. Honouring the timeout instead would need race state stored separately, only to always return { key: "timeout" }. Documented in the ctx.waitForAny() reference.

C23: a caught invokeAndWait timeout broke later child calls and was not memoized

Root cause. invokeAndWait treated "the claimed snapshot has waiting_on_execution_id" as "this call timed out". So a second invokeAndWait in the same attempt threw the same timeout without starting its child. The timeout itself was also never stored: once clearWaitingState had cleared the waiting fields, a later replay invoked the timed-out child a second time and waited on it again.

Fix. Following R4's note, clearing the snapshot alone is not enough; the timeout has to be stored.

  • The step of an invokeAndWait now stores {status: 'completed', result} (settlement) or {status: 'timed_out'}. The raw child result could not be told apart from a timeout marker.
  • clearWaitingState stores the timeout under the parent's waiting_step_key in the same statement that cancels the child and clears the wait. It is fenced by the claim token and returns whether it still held the claim. If not, the attempt hangs up as suspended instead of throwing.
  • invokeAndWait treats a missing result as a timeout only when the claim's waiting_step_key is its own key. A claim keeps that key only while the execution waits on a child. Any other key starts the child.

O4-08: conductor.waitForResult timeout during a blocked read

getExecution now cancels its query (postgres.js query.cancel()) when the signal aborts. waitForResult rejects with the abort reason (a TimeoutError for the timeout) whether the read fails because of the cancel or returns after the deadline. Cancelling on abort applies only to this read; settlement still runs under the worker's aborted signal, as O4-08 asked.

D5c: durations

ctx.sleep(), BatchTaskContext.sleep(), the ctx.invokeAndWait() timeout and the conductor.waitForResult() timeout now take a DurationInput (numbers are milliseconds, or a string such as "30s"), parsed with parseDuration(). The event and result waits already took one. Config fields whose name carries the unit (timeoutMs, pollIntervalMs, batch.timeoutMs, throttle.seconds, removeOnComplete.days) are unchanged.

Behaviour changes

  • timeout: 0 means zero, never infinity. Before, ctx.invokeAndWait(..., { timeout: 0 }) waited forever, and conductor.waitForResult(id, { timeout: 0 }) had no timeout. Now the first times out at once and cancels the child, and the second rejects with a TimeoutError at once. Omit timeout to wait indefinitely.
  • Negative, non-finite or malformed durations now throw (parseDuration). Before, they were passed to SQL or setTimeout.
  • A caught invokeAndWait timeout is thrown again on every replay, instead of invoking the child again.
  • waitForAny with no open branch throws instead of waiting.
  • Stored invokeAndWait steps change shape ({status, result}). The migration is edited in place, as the project does during active development. A database with child waits in flight from an older build would read their stored result wrongly.

Tests

tests/integration/wait-deadlines.test.ts brings in the R3/R4 reproductions with their barriers and fake time unchanged:

  • C11 (3 cases: completion, cancellation, and a mixed event/execution race): the waiter's orchestrator stops after it suspends, fake time moves past the deadline, then the awaited execution settles and a new orchestrator resumes the waiter. All 3 fail on main and pass now.
  • C12 (no branches, and only a branch an earlier race closed): the repro asserted "settles", which suited either fix. It now asserts the fail-fast error, with maxAttempts: 1.
  • C23 (both tests): one change from the repro. A memoized timeout is thrown again on each replay, so errors holds one entry per run of the handler, and the sleeping flag (also true for a completed parent) is no longer asserted.
  • O4-08: a second connection holds an access exclusive lock on the executions table. waitForResult(id, { timeout: 100 }) must reject with TimeoutError while the lock is still held, and the connection is usable afterwards. Without the change the read stays blocked.
  • D5c: invokeAndWait with timeout: 0 times out at once and cancels the child. conductor.waitForResult with timeout: 0 rejects. ctx.sleep("nap", "1m") reschedules by one minute.

The existing test for a first registration against an execution that has already settled still passes.

just lint, just format, pnpm typecheck and the full pnpm test pass. One existing test (wait-for-result.test.ts, "a pending child cancelled when its parent's wait times out") set waiting_on_execution_id by hand without waiting_step_key, a state settlement never produces; it now sets both.

Hot paths

  • In settlement, only the projection of parent_steps_all changes (it wraps the result in jsonb_build_object); the plan does not change.
  • _private_resolve_result_waits runs only when a settled execution has waiters. Its new join is a lookup on the unique (execution_id, step_key) index.
  • _private_register_event_race and clearWaitingState run once per wait, not per claim or settlement.

Follow-ups (out of scope)

  • invokeAndWait deadlines in settlement. A child that completes after the parent's deadline, but before the parent resumes, still delivers its result (completed_parents does not compare parent.run_at with now). A child that fails late still fails the parent through the cascade. Applying the same terminal-time rule there is a change to settlement and deserves its own PR.
  • conductor.waitForResult logs the cancelled read at error level through DatabaseClient.query() ("canceling statement due to user request"). It is harmless, but noisy if callers often time out against a blocked database.

psteinroe added a commit that referenced this pull request Oct 20, 2026
…nalWithStart and ctx.emit (#100)

Adds a `{ signal: name }` branch to `ctx.waitForAny()`. A keyed session
started with `signalWithStart` can now suspend until its next input or
its tool execution, whichever comes first, instead of ending:

```ts
ctx.onSignal("input", (payload, { id }) => saveInput(id, payload));
const winner = await ctx.waitForAny(`idle:${n}`, {
  input: { signal: "input" },            // new
  tool: { execution: toolExecutionId },
});
// { key: "input", signal: { id, name } } | { key: "tool", outcome } | { key: "timeout" }
```

**Registration and wake (SQL)**

A signal branch is an `execution_wait` subscription with a new
`signal_name` column. Its step key is `<stepKey>:<branch>`, the same as
execution branches. `_private_register_event_race` holds the claimed
execution's row lock. While holding it, the race wins at once if the
execution has an undelivered signal with that name; otherwise it
subscribes. Senders (`signal_execution`, `signal_with_start`, so also
`conductor.signal`, `ctx.sendSignal` and `ctx.signalWithStart`) already
lock that row and bump `signal_count`. Under that same lock,
`_private_store_signal` now does three things for a new signal: it
stores `{status: 'resolved', signal: {id, name}}` under the step key of
every unexpired matching subscription, removes those subscriptions, and
wakes the execution if it is suspended at a wait (the same condition
dispatch uses). Every signal is ordered against registration by the row
lock, so a signal stored between the check and the suspend wakes the
execution. No new lock is taken, and the sender still locks only the
signalled row, which is documented in the AGENTS.md lock-order section.

Signals for names no waiting branch listens for behave as before: they
are kept for the next attempt and do not wake the execution. Branches
are registered only on the first call, the same as execution branches in
#99. A signal that arrives after the deadline does not win (`expires_at
>= now`, matching #99's rule for result waits). Winner selection, the
timeout, closing losing branches and removing their subscriptions all
reuse the existing race code, so the one-winner guarantee from #79
holds.

**The branch does not consume the signal**

The signal stays undelivered. The next attempt primes it at claim and
delivers it to `ctx.onSignal()`. The result carries only the signal's `{
id, name }`.

**Deviation from the brief: signals being handled count too**

"Undelivered" alone leaves a lost wake-up. A handler can return and
acknowledge a signal after the user's loop checked its saved inputs but
before the race registers. The wait then suspends with an unprocessed
input and nothing to wake it. To close that gap, `SignalInbox.hold()`
stops new handlers from starting between the `waitForAny()` call and the
end of registration. The signal whose handler is still running at the
call is passed as `p_delivering_signal_id` and counts like an
undelivered one. The documented rule is: a signal wins unless its
handler returned before the call. Docs also say to register a handler
for every signal branch. Without one, the signal is never delivered, so
it wins every later wait at once.

**Ride-along: `metadata` for `ctx.signalWithStart` and `ctx.emit`**

Both now take `metadata?: Metadata | ((metadata) => Metadata)`, the same
option as `ctx.invoke` and `ctx.invokeAndWait`. It is resolved by the
same `childMetadata()` helper, so it is validated against the
Conductor's `metadata` schema and the 8 KB limit, and it defaults to
this execution's metadata. `ctx.signalWithStart` applies it only when
the call starts the execution, and the started id stays memoized under
the step. A running execution keeps its metadata. For `ctx.emit` it
becomes the event's metadata and flows to the executions the event
triggers. `ctx.sendSignal` starts nothing, so it gets no option, and the
docs say so. `tests/integration/execution-metadata.test.ts` covers an
override, the function form, inheritance by default, a signal to a
running execution that keeps its metadata, and schema-validation
failures for both. `execution-metadata-types.test.ts` covers the types.

**Verification**

New `tests/integration/wait-for-signal.test.ts`:
- a signal stored before the wait wins without suspending (handler
entered once)
- a signal whose handler is running at the call wins, and so does one
whose handler returns and is acknowledged while the step read is held
back. The second test fails if the hold starts after `loadStep`, and the
first fails without the delivering id.
- a signal wakes a suspended wait; replays return the memoized winner;
`onSignal` still receives the signal
- two deterministic races on separate connections with held
transactions: a signal blocked behind an uncommitted registration wakes
it, and a registration blocked behind an uncommitted signal wins at once
- one winner when a signal and an execution branch settle at the same
moment (two orchestrators)
- `signalWithStart` wakes a keyed session waiting on input or its tool
- a signal for another name does not wake the wait (`run_at` stays
`infinity`)
- under a set test clock: a signal wakes the wait. After advancing past
the timeout, a late signal does not win and the timeout does. This test
fails without the expiry check.

Unit tests cover `SignalInbox.hold()` and the `WaitForAnyResult` type
for signal branches. `just lint`, `just format`, `pnpm typecheck` and
the full `pnpm test` (579 tests, plus the new ones) pass.

**Follow-ups / interaction with open PRs**
- #98 (`steps.seq`): when it lands, both new step inserts (in
`_private_store_signal` and the race's immediate-signal path) need `seq`
from `_private_event_seq`. Without it, a signal and an execution
resolved at the same frozen time fall back to branch order. I'll add
that on rebase if #98 lands first.
- #99: compatible. Signal branches already follow its first-call-only
registration and deadline rules.
…d timeouts

- result waits store no outcome after their deadline (C11)
- waitForAny without an open branch raises instead of suspending forever (C12)
- a caught invokeAndWait timeout is memoized and no longer fails later child calls (C23)
- conductor.waitForResult cancels a blocked read on timeout (O4-08)
- durations take DurationInput everywhere; timeout: 0 means zero (D5c)
@psteinroe
psteinroe merged commit 4f93a84 into main Oct 20, 2026
11 checks passed
@psteinroe
psteinroe deleted the fix/wait-deadlines branch October 20, 2026 19:20
psteinroe added a commit that referenced this pull request Oct 20, 2026
… test clock (#98)

## What

Under a set test clock (`createTestClock`, `db.client.setFakeTime`),
event waits never received events emitted after them. Dispatch requires
`source.created_at > subscription.created_at`, and
`_private_current_time()` returned the frozen `fake_now` as-is, so the
subscription and the later event got the same timestamp and the event
was dropped. This affected `ctx.waitForEvent()`, `ctx.subscribe()` +
`ctx.waitForAny()`, and anything else that waits on events while the
clock is set. `waitForAny` also picked the wrong winner: it orders
resolved branches by step `created_at`, so ties fell back to branch
order.

## Fix: fake time keeps running

Production code and schema are unchanged; only fake time changes.

- `_private_current_time()` returns `clock_timestamp() +
pgconductor.fake_offset` when the setting is set. It replaces the
`pgconductor.fake_now` setting.
- `createTestClock`: `set(date)` stores `date - clock_timestamp()` as
the offset, `advance(d)` adds `d` to it, and `reset()` clears it.
`now()` applies the same offset to `Date.now()`, rounded up to whole
milliseconds so it never reads before the time set, so database and
worker time run on together (the worker's sampled offset stays ~0). The
offset is computed once in a materialized CTE. Without that, Postgres
inlines the subquery and `advance` reads the setting it just wrote.
- `db.client.setFakeTime()` / `clearFakeTime()` use the same setting.

I first pushed a fix that added a sequence (`_private_event_seq`, with
seq columns on subscriptions and steps, and in the dispatch payload). I
dropped it: it changed the production schema for a problem that only
exists in tests.

## Behaviour change (test clock)

**Fake time no longer stands still.** After `clock.set(date)` or
`advance()`, time keeps running from that point like real time. Tests
that compare exact timestamps after setting the clock need bounds
instead (`toBeGreaterThanOrEqual`). I updated the docs
(`api/testing.md`, `crafting-tasks/testing.md`) and AGENTS.md.

Tests updated for this change:
- `testing.test.ts`: exact `completedAt` and
`current_setting('pgconductor.fake_now')` checks now use bounds; added
"fake time keeps running from the time it was set to"
- `orchestrator-cleanup.test.ts`, `window-execution.test.ts`,
`dead-letter.test.ts`: exact timestamp checks now use bounds
- `batch-processing.test.ts`: the batch sleep test advanced exactly 1s
past a 1s sleep; it now uses 2s steps
- `wait-deadlines.test.ts` (from #99): the exact `ctx.sleep` deadline
check now uses bounds

## Verification

- New `wait-for-event` tests, each run with fake time and with real
time:
  - `waitForEvent` receives an event emitted after it
  - `subscribe` + `waitForAny` receive an event emitted after subscribe
  - `waitForAny` prefers the branch stored first
- an event emitted before the wait, and held back so it is dispatched
only after the subscription exists, is ignored
- The fake-time versions failed before the fix (the event was never
delivered); the real-time versions passed.
- `just lint`, `just format`, `pnpm typecheck`, `pnpm test` (612 tests)
pass.

## Follow-ups / notes

- With real time, the "after the subscription" check still relies on
`clock_timestamp()` being monotonic, so a wall-clock step (NTP) could
misorder an event and a subscription. That was already the case before
this PR.
- The index `idx_custom_event_subscription_wait_match (event_key,
created_at, expires_at, id)` does not seem to be used by any query
(dispatch joins subscriptions by id).
- `throttle-debounce` "debounce with partition keys" failed once in CI
on Postgres 15. It uses real time (no fake clock), and two invokes can
fall into different 60s debounce slots. It looks like an existing flake,
unrelated to this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant