[psteinroe]

Fixes the cancellation findings from the review: C13 (O2-01, O6-04), C14 (O2-02, O3-04, O6-05, O3-08), C16 (O3-05), and the two follow-ups #85 left open: an invokeAndWait timeout and a dedupe supersede did not cascade to linked children.

Root cause

Every path that cancels linked children found them in the snapshot of the statement that ended the parent, and then locked them in whatever order that statement happened to reach them:

• C13: settlement's linked_descendants CTE walked the graph from the settlement statement's snapshot. Suppose a child suspends on a new grandchild while the settlement waits for that child's lock. The settlement then fails the child but never sees the grandchild, and nothing cancels the grandchild later.
• C14:
  • The cascade locked descendants shallowest first, while child settlement locks child before parent.
  • cancel_execution locked a parent, then waited for its linked child inside the recursive call. This deadlocks when the parent waits for the child's result.
  • _private_register_event_race locked awaited executions by id, but one of them can be waiting on another.
  • drop_queue deleted tasks before it locked the queue row.
• C16 and the feat(tasks): ctx.invoke(..., { cancelWithParent: true }) #85 follow-ups: supersede in invoke() and clearWaitingState failed executions directly and never cascaded. clearWaitingState also did not set cancelled on a pending child.

Fix

One cancellation path, which reads the graph fresh and takes its locks all at once:

• _private_cancel_linked_children(parents) calls cancel_execution for the non-terminal linked children of ended executions, each with its parent's reason. It reads the children in a new statement, so its snapshot postdates the caller's locks. Callers:
  • Settlement, from the main query, after a writes CTE that counts the rows of every write CTE. The filter on that count runs all writes to completion before the cascade starts. The cascade then sees a linked child settled in the same batch in its new state: a retried, sleeping, re-queued or suspended child is cancelled, and a completed one stays completed. This replaces linked_descendants, cancelled_descendants, the three descendant writes and the six cancelled_descendants exclusions, so settlement now has 5 fewer references to the executions table.
  • cancel_execution, for every row it fails.
  • invoke(), for the execution it supersedes. The parent still ends as failed with superseded by reinvoke.
• clearWaitingState (the invokeAndWait timeout) cancels the awaited child with cancel_execution(child, 'Cancelled: parent timed out') before it clears the parent's wait. The child's waited-on chain and linked children are now cancelled too. A pending child is now cancelled = true, and its result waiters see {status: 'cancelled'} instead of {status: 'failed'}; this is the follow-up feat(tasks): ctx.invoke(..., { cancelWithParent: true }) #85 asked for. A running child now also gets last_error.
• _private_lock_executions(ids) locks a whole set with for update nowait. If a row is busy, it waits for each row alone in a subtransaction it rolls back, then tries the whole set again, so it never waits while holding part of the set. Callers:
  • cancel_execution, whose lock set now also includes the linked subtree of the rows it fails, so its recursive calls find their locks already held.
  • _private_register_event_race, for its awaited executions.
• drop_queue first locks its queue and the queues whose tasks dead-letter into it, in name order, the same order registration uses.

The lock order is now written down in AGENTS.md (Lock Order): awaited before waiter, parent before linked child, sets taken all at once, queue rows before tasks.

Where this differs from the review's suggestions:

• O2-02 / O6-05A suggested locking descendants deepest first. That alone does not fix C13, and cancel_execution's recursion still waits while holding the parent (O3-04). Routing the settlement cascade through cancel_execution fixes both. It also removes the same-batch exclusions, which existed only because the cascade ran in the same snapshot as the writes.
• O6-05B suggested nowait plus a client retry on 55P03. _private_lock_executions waits inside the function instead, so callers such as drop_queue or a direct cancel_execution never see the error.

Tests

The repro tests from review/repro-r2 moved into tests/integration/, with their barriers (pg_blocking_pids) kept. All of them failed on main and pass here:

• cancellation-cascades.test.ts:
  • C13: settlement with removeOnFail null and 0, plus the cancel_execution control.
  • C16: pending and suspended child, with removeOnFail null and 0.
  • New: an invokeAndWait timeout cancels the awaited child, its grandchild and its linked child.
  • New: a parent's failure settlement sees the linked children settled in the same batch in their new state (a sleeping one is cancelled, a completed one stays completed).
• lock-order.test.ts: the five C14 pairs. In the drop_queue test, registration now takes its lock first and drop_queue runs into it. A new variant covers a queue that dead-letters into the dropped one (O3-08's untested case). Both fail on main with 40P01.
• The shared session helpers are in tests/fixtures/sessions.ts.

Assertions updated for the intended behaviour change: a pending child cancelled by a timeout now has cancelled: true, and its waiters see cancelled (cancellation, invoke-support and wait-for-result tests). worker-lifecycle's fault injection now matches the new settlement text.

just lint, just format and pnpm typecheck pass, and the full pnpm test passes (560 tests, after rebasing onto #92).

Settlement performance

I seeded a fresh database per run: each queue holds the given number of rows (a third completed, a third failed, 20% parented). Each run settled batches of 100 freshly claimed rows on one queue. Planning and execution times come from pg_stat_statements with track_planning on Postgres 15:

• First call: one settlement on a new connection.
• Steady: mean over 40 calls on a warm connection.

	main plan / exec (ms)	branch plan / exec (ms)
6 queues × 50k, completed, first call	34.2 / 7.6	26.4 / 10.3
6 queues × 50k, completed, steady	0 / 9.1	0 / 9.8
6 queues × 50k, permanently failed, first call	30.7 / 7.2	22.2 / 9.3
6 queues × 50k, permanently failed, steady	0 / 5.4	0 / 6.7
106 queues × 5k, completed, first call	673 / 12506	498 / 18
106 queues × 5k, completed, steady	134 / 16029	0 / 21.6
106 queues × 5k, permanently failed, steady	86 / 16650	0 / 145

• Planning: first-call planning drops by about 25%, because settlement has 5 fewer references to the partitioned table.
• At 6 queues: steady execution is within about 1 ms.
• At 106 queues: EXPLAIN ANALYZE puts main's 12.5 s inside the failure-path CTEs even for an all-completed batch; that plan is gone here. One cost remains: a failure batch pays for _private_cancel_linked_children probing parent_execution_id in every partition, 145 ms for 100 failures at 106 queues. That is for F10, which plans to move rare cross-queue paths into functions anyway.

The harness was a throwaway script against a scratch container.

Follow-ups (out of scope)

• Locks that are still taken while holding others, which Postgres detects and which settlement retries:
  • A settlement delivering a result to a waiter that is running again, while that waiter's own failure settlement cascades to the awaited child.
  • _private_resolve_result_waits locking several waiters by id.
• _private_cancel_linked_children could skip partitions a queue's children never live in (F10).