Skip to content

fix(testing): keep fake time running so event waits match under a set test clock - #98

Merged
psteinroe merged 3 commits into
mainfrom
fix/test-clock-event-waits
Oct 20, 2026
Merged

psteinroe merged 3 commits into
mainfrom
fix/test-clock-event-waits

Conversation

@psteinroe

@psteinroe psteinroe commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

What

Under a set test clock (createTestClock, db.client.setFakeTime), event waits never received events emitted after them. Dispatch requires source.created_at > subscription.created_at, and _private_current_time() returned the frozen fake_now as-is, so the subscription and the later event got the same timestamp and the event was dropped. This affected ctx.waitForEvent(), ctx.subscribe() + ctx.waitForAny(), and anything else that waits on events while the clock is set. waitForAny also picked the wrong winner: it orders resolved branches by step created_at, so ties fell back to branch order.

Fix: fake time keeps running

Production code and schema are unchanged; only fake time changes.

  • _private_current_time() returns clock_timestamp() + pgconductor.fake_offset when the setting is set. It replaces the pgconductor.fake_now setting.
  • createTestClock: set(date) stores date - clock_timestamp() as the offset, advance(d) adds d to it, and reset() clears it. now() applies the same offset to Date.now(), rounded up to whole milliseconds so it never reads before the time set, so database and worker time run on together (the worker's sampled offset stays ~0). The offset is computed once in a materialized CTE. Without that, Postgres inlines the subquery and advance reads the setting it just wrote.
  • db.client.setFakeTime() / clearFakeTime() use the same setting.

I first pushed a fix that added a sequence (_private_event_seq, with seq columns on subscriptions and steps, and in the dispatch payload). I dropped it: it changed the production schema for a problem that only exists in tests.

Behaviour change (test clock)

Fake time no longer stands still. After clock.set(date) or advance(), time keeps running from that point like real time. Tests that compare exact timestamps after setting the clock need bounds instead (toBeGreaterThanOrEqual). I updated the docs (api/testing.md, crafting-tasks/testing.md) and AGENTS.md.

Tests updated for this change:

  • testing.test.ts: exact completedAt and current_setting('pgconductor.fake_now') checks now use bounds; added "fake time keeps running from the time it was set to"
  • orchestrator-cleanup.test.ts, window-execution.test.ts, dead-letter.test.ts: exact timestamp checks now use bounds
  • batch-processing.test.ts: the batch sleep test advanced exactly 1s past a 1s sleep; it now uses 2s steps
  • wait-deadlines.test.ts (from fix(waits): enforce wait deadlines, fail empty races and memoize child timeouts #99): the exact ctx.sleep deadline check now uses bounds

Verification

  • New wait-for-event tests, each run with fake time and with real time:
    • waitForEvent receives an event emitted after it
    • subscribe + waitForAny receive an event emitted after subscribe
    • waitForAny prefers the branch stored first
    • an event emitted before the wait, and held back so it is dispatched only after the subscription exists, is ignored
  • The fake-time versions failed before the fix (the event was never delivered); the real-time versions passed.
  • just lint, just format, pnpm typecheck, pnpm test (612 tests) pass.

Follow-ups / notes

  • With real time, the "after the subscription" check still relies on clock_timestamp() being monotonic, so a wall-clock step (NTP) could misorder an event and a subscription. That was already the case before this PR.
  • The index idx_custom_event_subscription_wait_match (event_key, created_at, expires_at, id) does not seem to be used by any query (dispatch joins subscriptions by id).
  • throttle-debounce "debounce with partition keys" failed once in CI on Postgres 15. It uses real time (no fake clock), and two invokes can fall into different 60s debounce slots. It looks like an existing flake, unrelated to this PR.

@psteinroe
psteinroe force-pushed the fix/test-clock-event-waits branch from 64b8fc9 to 1d1b6e3 Compare October 20, 2026 15:37
@psteinroe psteinroe changed the title fix(events): order event waits by sequence so they match under a frozen test clock fix(testing): keep fake time running so event waits match under a set test clock Oct 20, 2026
psteinroe added a commit that referenced this pull request Oct 20, 2026
…nalWithStart and ctx.emit (#100)

Adds a `{ signal: name }` branch to `ctx.waitForAny()`. A keyed session
started with `signalWithStart` can now suspend until its next input or
its tool execution, whichever comes first, instead of ending:

```ts
ctx.onSignal("input", (payload, { id }) => saveInput(id, payload));
const winner = await ctx.waitForAny(`idle:${n}`, {
  input: { signal: "input" },            // new
  tool: { execution: toolExecutionId },
});
// { key: "input", signal: { id, name } } | { key: "tool", outcome } | { key: "timeout" }
```

**Registration and wake (SQL)**

A signal branch is an `execution_wait` subscription with a new
`signal_name` column. Its step key is `<stepKey>:<branch>`, the same as
execution branches. `_private_register_event_race` holds the claimed
execution's row lock. While holding it, the race wins at once if the
execution has an undelivered signal with that name; otherwise it
subscribes. Senders (`signal_execution`, `signal_with_start`, so also
`conductor.signal`, `ctx.sendSignal` and `ctx.signalWithStart`) already
lock that row and bump `signal_count`. Under that same lock,
`_private_store_signal` now does three things for a new signal: it
stores `{status: 'resolved', signal: {id, name}}` under the step key of
every unexpired matching subscription, removes those subscriptions, and
wakes the execution if it is suspended at a wait (the same condition
dispatch uses). Every signal is ordered against registration by the row
lock, so a signal stored between the check and the suspend wakes the
execution. No new lock is taken, and the sender still locks only the
signalled row, which is documented in the AGENTS.md lock-order section.

Signals for names no waiting branch listens for behave as before: they
are kept for the next attempt and do not wake the execution. Branches
are registered only on the first call, the same as execution branches in
#99. A signal that arrives after the deadline does not win (`expires_at
>= now`, matching #99's rule for result waits). Winner selection, the
timeout, closing losing branches and removing their subscriptions all
reuse the existing race code, so the one-winner guarantee from #79
holds.

**The branch does not consume the signal**

The signal stays undelivered. The next attempt primes it at claim and
delivers it to `ctx.onSignal()`. The result carries only the signal's `{
id, name }`.

**Deviation from the brief: signals being handled count too**

"Undelivered" alone leaves a lost wake-up. A handler can return and
acknowledge a signal after the user's loop checked its saved inputs but
before the race registers. The wait then suspends with an unprocessed
input and nothing to wake it. To close that gap, `SignalInbox.hold()`
stops new handlers from starting between the `waitForAny()` call and the
end of registration. The signal whose handler is still running at the
call is passed as `p_delivering_signal_id` and counts like an
undelivered one. The documented rule is: a signal wins unless its
handler returned before the call. Docs also say to register a handler
for every signal branch. Without one, the signal is never delivered, so
it wins every later wait at once.

**Ride-along: `metadata` for `ctx.signalWithStart` and `ctx.emit`**

Both now take `metadata?: Metadata | ((metadata) => Metadata)`, the same
option as `ctx.invoke` and `ctx.invokeAndWait`. It is resolved by the
same `childMetadata()` helper, so it is validated against the
Conductor's `metadata` schema and the 8 KB limit, and it defaults to
this execution's metadata. `ctx.signalWithStart` applies it only when
the call starts the execution, and the started id stays memoized under
the step. A running execution keeps its metadata. For `ctx.emit` it
becomes the event's metadata and flows to the executions the event
triggers. `ctx.sendSignal` starts nothing, so it gets no option, and the
docs say so. `tests/integration/execution-metadata.test.ts` covers an
override, the function form, inheritance by default, a signal to a
running execution that keeps its metadata, and schema-validation
failures for both. `execution-metadata-types.test.ts` covers the types.

**Verification**

New `tests/integration/wait-for-signal.test.ts`:
- a signal stored before the wait wins without suspending (handler
entered once)
- a signal whose handler is running at the call wins, and so does one
whose handler returns and is acknowledged while the step read is held
back. The second test fails if the hold starts after `loadStep`, and the
first fails without the delivering id.
- a signal wakes a suspended wait; replays return the memoized winner;
`onSignal` still receives the signal
- two deterministic races on separate connections with held
transactions: a signal blocked behind an uncommitted registration wakes
it, and a registration blocked behind an uncommitted signal wins at once
- one winner when a signal and an execution branch settle at the same
moment (two orchestrators)
- `signalWithStart` wakes a keyed session waiting on input or its tool
- a signal for another name does not wake the wait (`run_at` stays
`infinity`)
- under a set test clock: a signal wakes the wait. After advancing past
the timeout, a late signal does not win and the timeout does. This test
fails without the expiry check.

Unit tests cover `SignalInbox.hold()` and the `WaitForAnyResult` type
for signal branches. `just lint`, `just format`, `pnpm typecheck` and
the full `pnpm test` (579 tests, plus the new ones) pass.

**Follow-ups / interaction with open PRs**
- #98 (`steps.seq`): when it lands, both new step inserts (in
`_private_store_signal` and the race's immediate-signal path) need `seq`
from `_private_event_seq`. Without it, a signal and an execution
resolved at the same frozen time fall back to branch order. I'll add
that on rebase if #98 lands first.
- #99: compatible. Signal branches already follow its first-call-only
registration and deadline rules.
@psteinroe
psteinroe force-pushed the fix/test-clock-event-waits branch from 1d1b6e3 to bd6e02c Compare October 20, 2026 19:15
@psteinroe
psteinroe force-pushed the fix/test-clock-event-waits branch from 0365333 to f43c991 Compare October 20, 2026 19:32
@psteinroe
psteinroe merged commit 9f450da into main Oct 20, 2026
11 checks passed
@psteinroe
psteinroe deleted the fix/test-clock-event-waits branch October 20, 2026 19:37
psteinroe added a commit that referenced this pull request Oct 21, 2026
…n per schedule (#102)

Fixes follow-ups 1, 12 and 19 from the handover (originally noted in #89
and #90).

**What**

1. **Dynamic schedule payloads (#89, #90).** `ctx.schedule(task, name, {
cron }, { userId })` lost its payload in two ways:
- `_private_schedule_next_occurrence()` inserted successors without
`payload` or `priority`, so from the second occurrence on the payload
was `null` and the priority was `0`.
- `taskEventFor` gave cron executions no payload at all, so even the
first occurrence's payload never reached the handler. A dynamic
occurrence arrived as `{ name: "<schedule name>" }`, which matches none
of the handler's event types.

Successors now copy `payload` and `priority`, as they already did
`group` and `metadata`. Occurrences of a dynamic schedule (`dynamic::`
dedupe keys) now run as `{ name: "pgconductor.invoke", payload }`. That
matches the type `ctx.schedule` checks the payload against, and the
worker validates the payload against the task's schema, as it does for
every invocation. Static cron triggers are unchanged: `{ name: "<trigger
name>" }`, no payload.

2. **Two schedule chains (#89).** If `ctx.schedule()` replaced a
schedule while a running occurrence of the old schedule was inserting
its successor, both chains survived. There were two ways this happened:
- **Sequential:** the replacement commits between the occurrence's claim
and its successor insert, and the successor is inserted anyway with the
old cron expression.
   - **Concurrent:** neither statement sees the other's uncommitted row.

   The fix also has two parts:
- The scheduling statement moved into `_private_schedule()`, a plpgsql
function next to `_private_unschedule()`. It takes the queue row lock
`for update`, then deletes and upserts in later statements. The
successor path already takes that lock `for share`, so the two are now
ordered.
- `_private_schedule_next_occurrence()` inserts no successor while a
live occurrence of the same schedule with a later scheduled time exists.
It stores that occurrence's id as the step instead. It compares the
dedupe key's timestamp, not `run_at`, because retries and sleeps move
`run_at`.

3. **Docs (#90).** `testing.md` said the next occurrence is scheduled
after each run. It is scheduled when an occurrence starts, before its
handler. `cron.md` now describes the event a dynamic occurrence receives
and what replacing a running schedule does. `AGENTS.md` "Cron Schedules"
is updated.

**Behaviour change:** handlers that told dynamic occurrences apart by
`event.name === "<schedule name>"` now see `"pgconductor.invoke"`. That
shape was never in the handler's type, and the payload was never
delivered.

**Verification**

Test first. All three new tests in `cron-scheduling.test.ts` failed on
`main`:
- *every occurrence of a dynamic schedule gets its payload and
priority:* fake clock, two drained occurrences. Got `{ name: "user-u1",
payload: undefined }`, and the successor had no payload or priority.
- *a successor scheduled after the replacement is not inserted:* left
two pending occurrences.
- *a replacement concurrent with the successor removes it:* holds the
successor's transaction open and waits until the replacement either
finishes or blocks on a lock. Left two pending occurrences.

The existing test *cron task with failure still reschedules next
execution* caught a first version that compared `run_at`; the dedupe key
timestamp fixes that.

`just lint` (oxlint plus typecheck) and `just format` are clean. `pnpm
test`: 614 of 615 pass. The one failure is in `wait-deadlines.test.ts`
(*… settled after the deadline > times out*), which fails the same way
on `main` at `9f450da` when run alone (2–3 of its 11 tests). See the
follow-ups.

**Follow-ups (out of scope)**

- `wait-deadlines.test.ts` is flaky on `main`. Since #98, fake time
keeps running, so the waiter's 1 s fake deadline (12:00:00 → 12:00:01)
can pass during setup before the test sets the clock forward. The test
then sees a completion instead of a timeout. This fits F13 (test suite).
- Cron successors still don't carry `trace_context`, so each occurrence
starts a new trace. I left that unchanged on purpose.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant