Skip to content

fix(maintenance): fail executions of tasks no worker runs anymore - #105

Open
psteinroe wants to merge 1 commit into
mainfrom
fix/unknown-task-executions
Open

psteinroe wants to merge 1 commit into
mainfrom
fix/unknown-task-executions

Conversation

@psteinroe

Copy link
Copy Markdown
Owner

What

The queue's maintenance task now fails pending executions of tasks that the queue's workers no longer register, after a one-day grace period. The error is task <key> is not registered on queue <queue>.

  • _private_register_worker() sets _private_tasks.unregistered_at on the queue's task rows that a registration leaves out, and clears it on the ones it registers. Worker registration already owns the queue's task set, and task rows are never deleted.
  • Maintenance fails available executions in two cases: the task has been unregistered for more than a day, or the task has no row at all (never registered, e.g. a typo in a SQL invoke) and the execution is more than a day old. Executions still locked by a replica of the previous version are left alone.
  • failUnregisteredExecutions claims these executions for a random claimant and settles them as permanently_failed with buildReturnExecutions, both in one transaction. They take the normal terminal path: dead-letter delivery, result waiters, workflows waiting in invokeAndWait, and linked children. Because nothing outside the transaction sees the claim, the worker's orphan release cannot interfere. Settlement now left-joins _private_tasks in task_configs so it can also settle executions of tasks without a row.

Why

Executions of a removed task stayed pending forever: no worker claims them, and nothing ever failed them. That includes retries, sleeping workflows, event deliveries and dynamic schedule occurrences.

Grace: a fixed UNREGISTERED_GRACE_DAYS = 1, with no option. During a rolling deploy, replicas of the previous version still run the task. A rollback registers the task again, which clears unregistered_at. A day covers both.

No registration: nothing is failed on a queue without a registration. Maintenance only runs on queues with a running worker, and that worker has registered first. A queue no worker polls yet (e.g. a dead-letter destination) keeps its executions pending.

Verification

New integration tests in tests/integration/maintenance-task.test.ts. The first and third failed before the fix:

  • A removed task's standalone execution and an invokeAndWait child both stay pending within the grace. After db.clock.advance("25h") they are failed with the error, each is dead-lettered, the waiting parent fails with Child execution failed: ..., and a ctx.waitForResult waiter receives { status: "failed", error }.
  • Executions of a task that is registered again (rollback) are kept.
  • An execution of a never-registered task is failed once it is a day old.

just lint, just format, pnpm typecheck and the full pnpm test (618 tests) all pass.

Follow-ups / notes

  • Two services sharing a queue, which is already unsupported, now also fail each other's executions. The horizontal scaling docs mention this.
  • Out of scope: settlement's locked_ancestors still inner-joins _private_tasks, so a waiting parent whose task has no row is not cascaded.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant