Repository navigation
fix(testing): keep fake time running so event waits match under a set test clock - #98
Merged
Merged
Conversation
psteinroe
force-pushed
the
fix/test-clock-event-waits
branch
from
October 20, 2026 15:37
64b8fc9 to
1d1b6e3
Compare
psteinroe
added a commit
that referenced
this pull request
Oct 20, 2026
…nalWithStart and ctx.emit (#100) Adds a `{ signal: name }` branch to `ctx.waitForAny()`. A keyed session started with `signalWithStart` can now suspend until its next input or its tool execution, whichever comes first, instead of ending: ```ts ctx.onSignal("input", (payload, { id }) => saveInput(id, payload)); const winner = await ctx.waitForAny(`idle:${n}`, { input: { signal: "input" }, // new tool: { execution: toolExecutionId }, }); // { key: "input", signal: { id, name } } | { key: "tool", outcome } | { key: "timeout" } ``` **Registration and wake (SQL)** A signal branch is an `execution_wait` subscription with a new `signal_name` column. Its step key is `<stepKey>:<branch>`, the same as execution branches. `_private_register_event_race` holds the claimed execution's row lock. While holding it, the race wins at once if the execution has an undelivered signal with that name; otherwise it subscribes. Senders (`signal_execution`, `signal_with_start`, so also `conductor.signal`, `ctx.sendSignal` and `ctx.signalWithStart`) already lock that row and bump `signal_count`. Under that same lock, `_private_store_signal` now does three things for a new signal: it stores `{status: 'resolved', signal: {id, name}}` under the step key of every unexpired matching subscription, removes those subscriptions, and wakes the execution if it is suspended at a wait (the same condition dispatch uses). Every signal is ordered against registration by the row lock, so a signal stored between the check and the suspend wakes the execution. No new lock is taken, and the sender still locks only the signalled row, which is documented in the AGENTS.md lock-order section. Signals for names no waiting branch listens for behave as before: they are kept for the next attempt and do not wake the execution. Branches are registered only on the first call, the same as execution branches in #99. A signal that arrives after the deadline does not win (`expires_at >= now`, matching #99's rule for result waits). Winner selection, the timeout, closing losing branches and removing their subscriptions all reuse the existing race code, so the one-winner guarantee from #79 holds. **The branch does not consume the signal** The signal stays undelivered. The next attempt primes it at claim and delivers it to `ctx.onSignal()`. The result carries only the signal's `{ id, name }`. **Deviation from the brief: signals being handled count too** "Undelivered" alone leaves a lost wake-up. A handler can return and acknowledge a signal after the user's loop checked its saved inputs but before the race registers. The wait then suspends with an unprocessed input and nothing to wake it. To close that gap, `SignalInbox.hold()` stops new handlers from starting between the `waitForAny()` call and the end of registration. The signal whose handler is still running at the call is passed as `p_delivering_signal_id` and counts like an undelivered one. The documented rule is: a signal wins unless its handler returned before the call. Docs also say to register a handler for every signal branch. Without one, the signal is never delivered, so it wins every later wait at once. **Ride-along: `metadata` for `ctx.signalWithStart` and `ctx.emit`** Both now take `metadata?: Metadata | ((metadata) => Metadata)`, the same option as `ctx.invoke` and `ctx.invokeAndWait`. It is resolved by the same `childMetadata()` helper, so it is validated against the Conductor's `metadata` schema and the 8 KB limit, and it defaults to this execution's metadata. `ctx.signalWithStart` applies it only when the call starts the execution, and the started id stays memoized under the step. A running execution keeps its metadata. For `ctx.emit` it becomes the event's metadata and flows to the executions the event triggers. `ctx.sendSignal` starts nothing, so it gets no option, and the docs say so. `tests/integration/execution-metadata.test.ts` covers an override, the function form, inheritance by default, a signal to a running execution that keeps its metadata, and schema-validation failures for both. `execution-metadata-types.test.ts` covers the types. **Verification** New `tests/integration/wait-for-signal.test.ts`: - a signal stored before the wait wins without suspending (handler entered once) - a signal whose handler is running at the call wins, and so does one whose handler returns and is acknowledged while the step read is held back. The second test fails if the hold starts after `loadStep`, and the first fails without the delivering id. - a signal wakes a suspended wait; replays return the memoized winner; `onSignal` still receives the signal - two deterministic races on separate connections with held transactions: a signal blocked behind an uncommitted registration wakes it, and a registration blocked behind an uncommitted signal wins at once - one winner when a signal and an execution branch settle at the same moment (two orchestrators) - `signalWithStart` wakes a keyed session waiting on input or its tool - a signal for another name does not wake the wait (`run_at` stays `infinity`) - under a set test clock: a signal wakes the wait. After advancing past the timeout, a late signal does not win and the timeout does. This test fails without the expiry check. Unit tests cover `SignalInbox.hold()` and the `WaitForAnyResult` type for signal branches. `just lint`, `just format`, `pnpm typecheck` and the full `pnpm test` (579 tests, plus the new ones) pass. **Follow-ups / interaction with open PRs** - #98 (`steps.seq`): when it lands, both new step inserts (in `_private_store_signal` and the race's immediate-signal path) need `seq` from `_private_event_seq`. Without it, a signal and an execution resolved at the same frozen time fall back to branch order. I'll add that on rebase if #98 lands first. - #99: compatible. Signal branches already follow its first-call-only registration and deadline rules.
psteinroe
force-pushed
the
fix/test-clock-event-waits
branch
from
October 20, 2026 19:15
1d1b6e3 to
bd6e02c
Compare
psteinroe
force-pushed
the
fix/test-clock-event-waits
branch
from
October 20, 2026 19:32
0365333 to
f43c991
Compare
psteinroe
added a commit
that referenced
this pull request
Oct 21, 2026
…n per schedule (#102) Fixes follow-ups 1, 12 and 19 from the handover (originally noted in #89 and #90). **What** 1. **Dynamic schedule payloads (#89, #90).** `ctx.schedule(task, name, { cron }, { userId })` lost its payload in two ways: - `_private_schedule_next_occurrence()` inserted successors without `payload` or `priority`, so from the second occurrence on the payload was `null` and the priority was `0`. - `taskEventFor` gave cron executions no payload at all, so even the first occurrence's payload never reached the handler. A dynamic occurrence arrived as `{ name: "<schedule name>" }`, which matches none of the handler's event types. Successors now copy `payload` and `priority`, as they already did `group` and `metadata`. Occurrences of a dynamic schedule (`dynamic::` dedupe keys) now run as `{ name: "pgconductor.invoke", payload }`. That matches the type `ctx.schedule` checks the payload against, and the worker validates the payload against the task's schema, as it does for every invocation. Static cron triggers are unchanged: `{ name: "<trigger name>" }`, no payload. 2. **Two schedule chains (#89).** If `ctx.schedule()` replaced a schedule while a running occurrence of the old schedule was inserting its successor, both chains survived. There were two ways this happened: - **Sequential:** the replacement commits between the occurrence's claim and its successor insert, and the successor is inserted anyway with the old cron expression. - **Concurrent:** neither statement sees the other's uncommitted row. The fix also has two parts: - The scheduling statement moved into `_private_schedule()`, a plpgsql function next to `_private_unschedule()`. It takes the queue row lock `for update`, then deletes and upserts in later statements. The successor path already takes that lock `for share`, so the two are now ordered. - `_private_schedule_next_occurrence()` inserts no successor while a live occurrence of the same schedule with a later scheduled time exists. It stores that occurrence's id as the step instead. It compares the dedupe key's timestamp, not `run_at`, because retries and sleeps move `run_at`. 3. **Docs (#90).** `testing.md` said the next occurrence is scheduled after each run. It is scheduled when an occurrence starts, before its handler. `cron.md` now describes the event a dynamic occurrence receives and what replacing a running schedule does. `AGENTS.md` "Cron Schedules" is updated. **Behaviour change:** handlers that told dynamic occurrences apart by `event.name === "<schedule name>"` now see `"pgconductor.invoke"`. That shape was never in the handler's type, and the payload was never delivered. **Verification** Test first. All three new tests in `cron-scheduling.test.ts` failed on `main`: - *every occurrence of a dynamic schedule gets its payload and priority:* fake clock, two drained occurrences. Got `{ name: "user-u1", payload: undefined }`, and the successor had no payload or priority. - *a successor scheduled after the replacement is not inserted:* left two pending occurrences. - *a replacement concurrent with the successor removes it:* holds the successor's transaction open and waits until the replacement either finishes or blocks on a lock. Left two pending occurrences. The existing test *cron task with failure still reschedules next execution* caught a first version that compared `run_at`; the dedupe key timestamp fixes that. `just lint` (oxlint plus typecheck) and `just format` are clean. `pnpm test`: 614 of 615 pass. The one failure is in `wait-deadlines.test.ts` (*… settled after the deadline > times out*), which fails the same way on `main` at `9f450da` when run alone (2–3 of its 11 tests). See the follow-ups. **Follow-ups (out of scope)** - `wait-deadlines.test.ts` is flaky on `main`. Since #98, fake time keeps running, so the waiter's 1 s fake deadline (12:00:00 → 12:00:01) can pass during setup before the test sets the clock forward. The test then sees a completion instead of a timeout. This fits F13 (test suite). - Cron successors still don't carry `trace_context`, so each occurrence starts a new trace. I left that unchanged on purpose. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Under a set test clock (
createTestClock,db.client.setFakeTime), event waits never received events emitted after them. Dispatch requiressource.created_at > subscription.created_at, and_private_current_time()returned the frozenfake_nowas-is, so the subscription and the later event got the same timestamp and the event was dropped. This affectedctx.waitForEvent(),ctx.subscribe()+ctx.waitForAny(), and anything else that waits on events while the clock is set.waitForAnyalso picked the wrong winner: it orders resolved branches by stepcreated_at, so ties fell back to branch order.Fix: fake time keeps running
Production code and schema are unchanged; only fake time changes.
_private_current_time()returnsclock_timestamp() + pgconductor.fake_offsetwhen the setting is set. It replaces thepgconductor.fake_nowsetting.createTestClock:set(date)storesdate - clock_timestamp()as the offset,advance(d)addsdto it, andreset()clears it.now()applies the same offset toDate.now(), rounded up to whole milliseconds so it never reads before the time set, so database and worker time run on together (the worker's sampled offset stays ~0). The offset is computed once in a materialized CTE. Without that, Postgres inlines the subquery andadvancereads the setting it just wrote.db.client.setFakeTime()/clearFakeTime()use the same setting.I first pushed a fix that added a sequence (
_private_event_seq, with seq columns on subscriptions and steps, and in the dispatch payload). I dropped it: it changed the production schema for a problem that only exists in tests.Behaviour change (test clock)
Fake time no longer stands still. After
clock.set(date)oradvance(), time keeps running from that point like real time. Tests that compare exact timestamps after setting the clock need bounds instead (toBeGreaterThanOrEqual). I updated the docs (api/testing.md,crafting-tasks/testing.md) and AGENTS.md.Tests updated for this change:
testing.test.ts: exactcompletedAtandcurrent_setting('pgconductor.fake_now')checks now use bounds; added "fake time keeps running from the time it was set to"orchestrator-cleanup.test.ts,window-execution.test.ts,dead-letter.test.ts: exact timestamp checks now use boundsbatch-processing.test.ts: the batch sleep test advanced exactly 1s past a 1s sleep; it now uses 2s stepswait-deadlines.test.ts(from fix(waits): enforce wait deadlines, fail empty races and memoize child timeouts #99): the exactctx.sleepdeadline check now uses boundsVerification
wait-for-eventtests, each run with fake time and with real time:waitForEventreceives an event emitted after itsubscribe+waitForAnyreceive an event emitted after subscribewaitForAnyprefers the branch stored firstjust lint,just format,pnpm typecheck,pnpm test(612 tests) pass.Follow-ups / notes
clock_timestamp()being monotonic, so a wall-clock step (NTP) could misorder an event and a subscription. That was already the case before this PR.idx_custom_event_subscription_wait_match (event_key, created_at, expires_at, id)does not seem to be used by any query (dispatch joins subscriptions by id).throttle-debounce"debounce with partition keys" failed once in CI on Postgres 15. It uses real time (no fake clock), and two invokes can fall into different 60s debounce slots. It looks like an existing flake, unrelated to this PR.