Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The queue's maintenance task now fails pending executions of tasks that the queue's workers no longer register, after a one-day grace period. The error is
task <key> is not registered on queue <queue>._private_register_worker()sets_private_tasks.unregistered_aton the queue's task rows that a registration leaves out, and clears it on the ones it registers. Worker registration already owns the queue's task set, and task rows are never deleted.invoke) and the execution is more than a day old. Executions still locked by a replica of the previous version are left alone.failUnregisteredExecutionsclaims these executions for a random claimant and settles them aspermanently_failedwithbuildReturnExecutions, both in one transaction. They take the normal terminal path: dead-letter delivery, result waiters, workflows waiting ininvokeAndWait, and linked children. Because nothing outside the transaction sees the claim, the worker's orphan release cannot interfere. Settlement now left-joins_private_tasksintask_configsso it can also settle executions of tasks without a row.Why
Executions of a removed task stayed pending forever: no worker claims them, and nothing ever failed them. That includes retries, sleeping workflows, event deliveries and dynamic schedule occurrences.
Grace: a fixed
UNREGISTERED_GRACE_DAYS = 1, with no option. During a rolling deploy, replicas of the previous version still run the task. A rollback registers the task again, which clearsunregistered_at. A day covers both.No registration: nothing is failed on a queue without a registration. Maintenance only runs on queues with a running worker, and that worker has registered first. A queue no worker polls yet (e.g. a dead-letter destination) keeps its executions pending.
Verification
New integration tests in
tests/integration/maintenance-task.test.ts. The first and third failed before the fix:invokeAndWaitchild both stay pending within the grace. Afterdb.clock.advance("25h")they are failed with the error, each is dead-lettered, the waiting parent fails withChild execution failed: ..., and actx.waitForResultwaiter receives{ status: "failed", error }.just lint,just format,pnpm typecheckand the fullpnpm test(618 tests) all pass.Follow-ups / notes
locked_ancestorsstill inner-joins_private_tasks, so a waiting parent whose task has no row is not cascaded.