Repository navigation
fix(worker): stop() waits for signal acks and concurrent steps - #101
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
orchestrator.stop()could return while a handler was still running inside a step. That breaks the shutdown barrier from #94. This PR fixes two independent causes. Both also settled the attempt with code from it still running, outside of shutdown too.1. A signal acknowledgement that stop() ends
The inbox acknowledges signals with the worker's shutdown signal.
query()never cancels a statement that is already running: an acknowledgement that is merely in flight finishes, and stop waits as it should. The signal does stop retries, though. When stop() lands while an acknowledgement is retrying a connection error,query()throws. The inbox reports that assignal-failed, the attempt is failed while the handler still runs, and stop() returns at once. Actx.onSignalhandler that throws becausectx.signalaborted during shutdown takes the same path.Fix: once the worker signal has aborted, a delivery failure no longer aborts the attempt. The handler keeps its slot, and the attempt is settled as
releasedonce it returns, without using up a retry. The signal stays undelivered, so the next claim primes it again.2. Concurrent steps
A handler can run steps concurrently (
Promise.all/allSettled). Say one step's callback is still running and a sibling does one of these:step()during stop(),ctx.sleep(), a wait, a fenced saveThen the worker settled the attempt at once. stop() returned, and the running callback overlapped the next attempt.
Fix:
TaskContexttracks each step's callback and save (TaskContextOptions.steps). Before the worker reports the attempt and removes it from_running, it waits for the steps still in flight, so they save their results while the claim is still held. A save that comes after settlement is still fenced by the claim token and hangs up assuspended(C24/#94), so it never counts as a success. A timeout, cancellation or signal failure the worker initiates still ends the attempt at once, as it does for a sequential handler blocked in a step.The brief asked for the wait to be bounded by the #94 stop/cleanup deadline. I bounded it by
timeoutMsinstead. The #94 deadlines bound database calls (final flush, cleanup), not handler code. A sequential handler blocked in a step already holds stop() for as long as it runs, so a concurrent one now behaves the same.timeoutMsremains the per-attempt bound, including while the worker waits for steps.mapConcurrentnow joins its remaining mappers when a mapper throws or the consumer exits early. The worker's mapper only throws on an internal assert, so this path is defensive.Other worker calls that use the shutdown signal
I checked the other worker-side calls that use the shutdown signal. None of them turns a shutdown abort into an attempt failure:
scheduleNextCronExecutionand everyTaskContextstatement reject into the handler, andcatchreleases the attempt when the worker signal has aborted (same inexecuteBatch).registerWorker, the clock sampling and the fetch loop do not belong to an attempt.Tests
All in
tests/integration/worker-lifecycle.test.tsunless noted. Faults go through asqlproxy.stop() while a handler runs a step: a no-signal control, a slow acknowledgement, an acknowledgement held in flight during stop() (these three pass onmain); an acknowledgement retrying connection errors during stop(), and a signal handler rejecting onctx.signalduring stop() (these two fail onmain). After the fix: stop() waits, the execution is released withattempts = 0, and the signal stays undelivered.a step callback still running when a sibling ends the attempt: the sibling's checkpoint releases during stop(), a siblingctx.sleep(), a sibling that throws (these three fail onmain), andtimeoutMsbounding the wait. stop() and settlement now wait for the running callback, and its step is saved.tests/unit/lib/map-concurrent.test.ts: joins mappers on a mapper error and on an early exit (both fail onmain).just lint,just format,pnpm typecheckand the fullpnpm test(580 tests) all pass.Follow-ups
ctx.onSignalhandlers still running are not awaited. The inbox stops after the running handler but is closed withoutawait.