Skip to content

fix(worker): stop() waits for signal acks and concurrent steps - #101

Merged
psteinroe merged 2 commits into
mainfrom
fix/stop-signal-ack
Oct 20, 2026
Merged

psteinroe merged 2 commits into
mainfrom
fix/stop-signal-ack

Conversation

@psteinroe

@psteinroe psteinroe commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

orchestrator.stop() could return while a handler was still running inside a step. That breaks the shutdown barrier from #94. This PR fixes two independent causes. Both also settled the attempt with code from it still running, outside of shutdown too.

1. A signal acknowledgement that stop() ends

The inbox acknowledges signals with the worker's shutdown signal. query() never cancels a statement that is already running: an acknowledgement that is merely in flight finishes, and stop waits as it should. The signal does stop retries, though. When stop() lands while an acknowledgement is retrying a connection error, query() throws. The inbox reports that as signal-failed, the attempt is failed while the handler still runs, and stop() returns at once. A ctx.onSignal handler that throws because ctx.signal aborted during shutdown takes the same path.

Fix: once the worker signal has aborted, a delivery failure no longer aborts the attempt. The handler keeps its slot, and the attempt is settled as released once it returns, without using up a retry. The signal stays undelivered, so the next claim primes it again.

2. Concurrent steps

A handler can run steps concurrently (Promise.all/allSettled). Say one step's callback is still running and a sibling does one of these:

  • hangs up: the checkpoint in step() during stop(), ctx.sleep(), a wait, a fenced save
  • throws

Then the worker settled the attempt at once. stop() returned, and the running callback overlapped the next attempt.

Fix: TaskContext tracks each step's callback and save (TaskContextOptions.steps). Before the worker reports the attempt and removes it from _running, it waits for the steps still in flight, so they save their results while the claim is still held. A save that comes after settlement is still fenced by the claim token and hangs up as suspended (C24/#94), so it never counts as a success. A timeout, cancellation or signal failure the worker initiates still ends the attempt at once, as it does for a sequential handler blocked in a step.

The brief asked for the wait to be bounded by the #94 stop/cleanup deadline. I bounded it by timeoutMs instead. The #94 deadlines bound database calls (final flush, cleanup), not handler code. A sequential handler blocked in a step already holds stop() for as long as it runs, so a concurrent one now behaves the same. timeoutMs remains the per-attempt bound, including while the worker waits for steps.

mapConcurrent now joins its remaining mappers when a mapper throws or the consumer exits early. The worker's mapper only throws on an internal assert, so this path is defensive.

Other worker calls that use the shutdown signal

I checked the other worker-side calls that use the shutdown signal. None of them turns a shutdown abort into an attempt failure:

  • scheduleNextCronExecution and every TaskContext statement reject into the handler, and catch releases the attempt when the worker signal has aborted (same in executeBatch).
  • registerWorker, the clock sampling and the fetch loop do not belong to an attempt.
  • The final flush already retries with its own 10s deadline.

Tests

All in tests/integration/worker-lifecycle.test.ts unless noted. Faults go through a sql proxy.

  • stop() while a handler runs a step: a no-signal control, a slow acknowledgement, an acknowledgement held in flight during stop() (these three pass on main); an acknowledgement retrying connection errors during stop(), and a signal handler rejecting on ctx.signal during stop() (these two fail on main). After the fix: stop() waits, the execution is released with attempts = 0, and the signal stays undelivered.
  • a step callback still running when a sibling ends the attempt: the sibling's checkpoint releases during stop(), a sibling ctx.sleep(), a sibling that throws (these three fail on main), and timeoutMs bounding the wait. stop() and settlement now wait for the running callback, and its step is saved.
  • tests/unit/lib/map-concurrent.test.ts: joins mappers on a mapper error and on an early exit (both fail on main).

just lint, just format, pnpm typecheck and the full pnpm test (580 tests) all pass.

Follow-ups

  • A cancellation that arrives while the worker waits for concurrent steps after a hangup takes effect only once they settle. The attempt controller is already aborted, so settlement then fails the cancelled row.
  • After a hangup, ctx.onSignal handlers still running are not awaited. The inbox stops after the running handler but is closed without await.
  • An in-flight statement is still never cancelled on abort (already noted in fix(worker): attempt lifecycle and shutdown #94).

@psteinroe psteinroe changed the title fix(worker): release an attempt whose signal delivery fails during stop() fix(worker): stop() waits for signal acks and concurrent steps Oct 20, 2026
@psteinroe
psteinroe merged commit 907f5d5 into main Oct 20, 2026
11 checks passed
@psteinroe
psteinroe deleted the fix/stop-signal-ack branch October 20, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant