diff --git a/AGENTS.md b/AGENTS.md index af872b8..e51894f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -179,6 +179,7 @@ Tables live in the `pgconductor` schema with a `_private_` prefix (`_private_exe **Important**: All core logic lives in Postgres. The TypeScript layer is intentionally thin - it validates input and orchestrates calls via `DatabaseClient`. - Claiming (`buildGetExecutions`) and settlement (`buildReturnExecutions`) are single statements in `query-builder.ts` +- Settlement settles executions of one queue (`returnExecutions` takes the worker's `queueName`) and narrows every reference to the executions table to its partition. The queue is inlined as a literal, because a cached generic plan prunes partitions only by constants: each queue gets its own prepared statement, planned once per connection, whatever the number of queues. Reads of executions in other queues (waiting parents) run in functions called only when a settled execution has a parent: `_private_complete_children()` wakes the parents of completed children and returns the children no parent waits on, `_private_fail_waiting_parents()` fails waiting ancestors. Their statements reach every partition, so they force generic plans, made once per connection, and look rows up by arrays of ids to stay on indexes. They read with fresh snapshots, so they come first in `writes`. A new reference to executions in settlement must be narrowed to the queue or go into such a function - SQL functions in the migration: `invoke()`, `invoke_batch()`, `cancel_execution()`, `signal_execution()`, `signal_with_start()`, `emit_event()`, `drop_queue()`, `_private_register_worker()`, `_private_current_time()`, `_private_portable_uuidv7()`, ... ### Key Design Patterns @@ -198,7 +199,7 @@ Tables live in the `pgconductor` schema with a `_private_` prefix (`_private_exe **Cascade Failures** - When a child fails permanently (attempts >= max_attempts), its parent fails too -- Implemented in `buildReturnExecutions()` via the `permanently_failed_children` CTE +- Implemented by `_private_fail_waiting_parents()`, which settlement calls for the `permanently_failed_children` that have a parent - Parent receives error like "Child execution failed: " **Claims and Retries** @@ -273,7 +274,7 @@ Tables live in the `pgconductor` schema with a `_private_` prefix (`_private_exe - Every path that ends an execution (settlement's `resolved_result_waits`, `cancel_execution()`, `invoke()` superseding a running execution, `clearWaitingState`, and removing pending schedule executions) reads `result_waiters` from the locked or deleted row, like `signal_count`, so a wait registered after its snapshot is not missed. It passes the waits with their outcomes to `_private_resolve_result_waits`, which stores `{status: 'resolved', outcome}` under each wait's step key and wakes a waiter suspended at a wait, like dispatch does. A new path that ends or removes executions must do the same; `drop_queue()` does not - `_private_resolve_result_waits` wakes suspended waiters with its own update, which runs before the data-modifying CTEs of the calling statement. A statement that also updates or deletes a suspended execution must exclude it from the waits it passes, or the later update of that row fails or is lost - A race that settles removes its entries from the `result_waiters` of awaited executions still running, so timeouts and lost races do not accumulate entries -- The delivery is a plpgsql function called only when a settled execution has waiters: settlement is planned on every call, and each reference to the partitioned executions table adds planning cost per queue +- The delivery is a plpgsql function called only when a settled execution has waiters, so settlement does not plan its references to executions of other queues - Waiting creates no parent link: a failed or cancelled execution never fails its waiter, and timeouts or cancelling the waiter leave it running. An unknown execution raises at registration **Signals** @@ -344,7 +345,7 @@ Guidelines for SQL: Common issues: - **Timing issues**: Use fake time instead of waiting (backoff schedule is 15s, 30s, 60s...) -- **Cascade failures**: Check `permanently_failed_children` CTE logic +- **Cascade failures**: Check the `permanently_failed_children` CTE and `_private_fail_waiting_parents()` - **Infinity serialization**: postgres.js serializes infinity as null in JSON Inspect database state in tests: diff --git a/migrations/0000000001_setup.sql b/migrations/0000000001_setup.sql index 6c41804..737332a 100644 --- a/migrations/0000000001_setup.sql +++ b/migrations/0000000001_setup.sql @@ -1248,6 +1248,151 @@ begin end; $function$; +-- settlement of completed children ([{execution_id, result}]) that have a parent: stores the result +-- for the parents waiting on them and wakes those parents. returns the children no execution waits +-- on any more. parents may be in any queue, so settlement calls this only when needed instead of +-- planning these lookups on every call. they cannot be narrowed to a partition, so their plans are +-- generic and made once per connection. the input is estimated at 100 rows, so lookups use arrays +-- of ids to stay on indexes. +create or replace function pgconductor._private_complete_children(p_children jsonb) +returns setof uuid +language plpgsql +volatile +set search_path to '' +set plan_cache_mode to force_generic_plan +as $function$ +declare + v_ids uuid[] := array( + select c.execution_id from jsonb_to_recordset(p_children) as c(execution_id uuid) + ); + v_orphans uuid[]; +begin + v_orphans := array( + select unnest(v_ids) + except + select parent.waiting_on_execution_id + from pgconductor._private_executions parent + where parent.waiting_on_execution_id = any(v_ids) + ); + + -- a completed child may wake only a parent which is still waiting and is not + -- currently claimed. the row lock makes this check race-safe. + with parents as materialized ( + select parent.id, parent.queue, parent.waiting_step_key, c.result + from pgconductor._private_executions parent + join jsonb_to_recordset(p_children) as c(execution_id uuid, result jsonb) + on c.execution_id = parent.waiting_on_execution_id + where parent.waiting_on_execution_id = any(v_ids) + and parent.completed_at is null + and parent.failed_at is null + and parent.locked_by is null + for update of parent + ), + parent_steps as ( + insert into pgconductor._private_steps (execution_id, queue, key, result) + select p.id, p.queue, p.waiting_step_key, + jsonb_build_object('status', 'completed', 'result', p.result) + from parents p + where p.waiting_step_key is not null + on conflict (execution_id, key) do nothing + ) + update pgconductor._private_executions e + set run_at = pgconductor._private_current_time(), waiting_on_execution_id = null, + waiting_step_key = null, locked_by = null, locked_at = null + where e.id = any(array(select p.id from parents p)); + + return query select unnest(v_orphans); +end; +$function$; + +-- settlement of terminally failed children ([{execution_id, error, cancelled}]) that have a parent: +-- fails every execution still waiting on them, and the ones waiting on those, and returns them for +-- dead-lettering, result waits and the cancel cascade. like _private_complete_children, settlement +-- calls this only when needed. +create or replace function pgconductor._private_fail_waiting_parents(p_children jsonb) +returns table ( + execution_id uuid, queue text, task_key text, execution_group text, payload jsonb, + failure_error text, failure_attempts integer, execution_cancelled boolean, + dead_letter_queue text, dead_letter_task_key text, trace_context jsonb, metadata jsonb, + result_waiters jsonb +) +language plpgsql +volatile +set search_path to '' +set plan_cache_mode to force_generic_plan +as $function$ +declare + v_ids uuid[] := array( + select c.execution_id from jsonb_to_recordset(p_children) as c(execution_id uuid) + ); + v_ancestors uuid[] := '{}'; +begin + loop + v_ids := array( + select parent.id + from pgconductor._private_executions parent + where parent.waiting_on_execution_id = any(v_ids) + and parent.completed_at is null and parent.failed_at is null and parent.locked_by is null + ); + exit when cardinality(v_ids) = 0; + v_ancestors := v_ancestors || v_ids; + end loop; + + -- ancestors are locked child first, then the chain is followed again on the locked rows + return query + with recursive locked_ancestors as materialized ( + select parent.id, parent.queue, parent.waiting_on_execution_id, parent.task_key, + parent."group", parent.payload, parent.attempts, parent.trace_context, parent.metadata, + parent.result_waiters, + t.remove_on_fail_days = 0 as should_remove, + t.dead_letter_queue, t.dead_letter_task_key + from pgconductor._private_executions parent + join pgconductor._private_tasks t on t.key = parent.task_key and t.queue = parent.queue + where parent.id = any(v_ancestors) + and parent.completed_at is null and parent.failed_at is null and parent.locked_by is null + order by array_position(v_ancestors, parent.id) + for update of parent + ), + targets as ( + select c.error, c.cancelled, a.* + from jsonb_to_recordset(p_children) as c(execution_id uuid, error text, cancelled boolean) + join locked_ancestors a on a.waiting_on_execution_id = c.execution_id + union all + select t.error, t.cancelled, a.* + from targets t + join locked_ancestors a on a.waiting_on_execution_id = t.id + ), + removed as ( + delete from pgconductor._private_executions e + using targets t + where e.id = any(v_ancestors) and e.id = t.id and e.queue = t.queue + and t.should_remove and e.locked_by is null + ), + failed as ( + update pgconductor._private_executions e + set failed_at = pgconductor._private_current_time(), + last_error = 'Child execution failed: ' || t.error, + waiting_on_execution_id = null, waiting_step_key = null, + locked_by = null, locked_at = null + from targets t + where e.id = any(v_ancestors) and e.id = t.id and e.queue = t.queue + and t.should_remove is not true + returning e.id, e.queue + ), + deleted_waits as ( + delete from pgconductor._private_custom_event_subscriptions subscription + using failed f + where subscription.kind = 'execution_wait' + and subscription.execution_id = f.id + and subscription.queue = f.queue + ) + select t.id, t.queue, t.task_key, t."group", t.payload, 'Child execution failed: ' || t.error, + t.attempts, t.cancelled, t.dead_letter_queue, t.dead_letter_task_key, t.trace_context, + t.metadata, t.result_waiters + from targets t; +end; +$function$; + -- cancel an execution create or replace function pgconductor.cancel_execution( p_execution_id uuid, diff --git a/packages/pgconductor-js/src/database-client.ts b/packages/pgconductor-js/src/database-client.ts index 5bde58d..49f7798 100644 --- a/packages/pgconductor-js/src/database-client.ts +++ b/packages/pgconductor-js/src/database-client.ts @@ -133,12 +133,13 @@ export type ExecutionResult = export type ReturnExecutionsArgs = { orchestratorId: string; + /** The queue of every settled execution */ + queueName: string; results: ExecutionResult[]; }; export type ExecutionCompleted = { execution_id: string; - queue: string; task_key: string; claim_token: string; status: "completed"; @@ -149,7 +150,6 @@ export type ExecutionCompleted = { export type ExecutionFailed = { execution_id: string; - queue: string; task_key: string; claim_token: string; status: "failed"; @@ -158,7 +158,6 @@ export type ExecutionFailed = { export type ExecutionReleased = { execution_id: string; - queue: string; task_key: string; claim_token: string; status: "released"; @@ -168,7 +167,6 @@ export type ExecutionReleased = { export type ExecutionPermamentlyFailed = { execution_id: string; - queue: string; task_key: string; claim_token: string; status: "permanently_failed"; @@ -178,7 +176,6 @@ export type ExecutionPermamentlyFailed = { export type ExecutionInvokeChild = { group?: string | null; execution_id: string; - queue: string; task_key: string; claim_token: string; status: "invoke_child"; diff --git a/packages/pgconductor-js/src/generated/sql.ts b/packages/pgconductor-js/src/generated/sql.ts index 95d771b..4d6f905 100644 --- a/packages/pgconductor-js/src/generated/sql.ts +++ b/packages/pgconductor-js/src/generated/sql.ts @@ -1262,6 +1262,151 @@ begin end; $function$; +-- settlement of completed children ([{execution_id, result}]) that have a parent: stores the result +-- for the parents waiting on them and wakes those parents. returns the children no execution waits +-- on any more. parents may be in any queue, so settlement calls this only when needed instead of +-- planning these lookups on every call. they cannot be narrowed to a partition, so their plans are +-- generic and made once per connection. the input is estimated at 100 rows, so lookups use arrays +-- of ids to stay on indexes. +create or replace function pgconductor._private_complete_children(p_children jsonb) +returns setof uuid +language plpgsql +volatile +set search_path to '' +set plan_cache_mode to force_generic_plan +as $function$ +declare + v_ids uuid[] := array( + select c.execution_id from jsonb_to_recordset(p_children) as c(execution_id uuid) + ); + v_orphans uuid[]; +begin + v_orphans := array( + select unnest(v_ids) + except + select parent.waiting_on_execution_id + from pgconductor._private_executions parent + where parent.waiting_on_execution_id = any(v_ids) + ); + + -- a completed child may wake only a parent which is still waiting and is not + -- currently claimed. the row lock makes this check race-safe. + with parents as materialized ( + select parent.id, parent.queue, parent.waiting_step_key, c.result + from pgconductor._private_executions parent + join jsonb_to_recordset(p_children) as c(execution_id uuid, result jsonb) + on c.execution_id = parent.waiting_on_execution_id + where parent.waiting_on_execution_id = any(v_ids) + and parent.completed_at is null + and parent.failed_at is null + and parent.locked_by is null + for update of parent + ), + parent_steps as ( + insert into pgconductor._private_steps (execution_id, queue, key, result) + select p.id, p.queue, p.waiting_step_key, + jsonb_build_object('status', 'completed', 'result', p.result) + from parents p + where p.waiting_step_key is not null + on conflict (execution_id, key) do nothing + ) + update pgconductor._private_executions e + set run_at = pgconductor._private_current_time(), waiting_on_execution_id = null, + waiting_step_key = null, locked_by = null, locked_at = null + where e.id = any(array(select p.id from parents p)); + + return query select unnest(v_orphans); +end; +$function$; + +-- settlement of terminally failed children ([{execution_id, error, cancelled}]) that have a parent: +-- fails every execution still waiting on them, and the ones waiting on those, and returns them for +-- dead-lettering, result waits and the cancel cascade. like _private_complete_children, settlement +-- calls this only when needed. +create or replace function pgconductor._private_fail_waiting_parents(p_children jsonb) +returns table ( + execution_id uuid, queue text, task_key text, execution_group text, payload jsonb, + failure_error text, failure_attempts integer, execution_cancelled boolean, + dead_letter_queue text, dead_letter_task_key text, trace_context jsonb, metadata jsonb, + result_waiters jsonb +) +language plpgsql +volatile +set search_path to '' +set plan_cache_mode to force_generic_plan +as $function$ +declare + v_ids uuid[] := array( + select c.execution_id from jsonb_to_recordset(p_children) as c(execution_id uuid) + ); + v_ancestors uuid[] := '{}'; +begin + loop + v_ids := array( + select parent.id + from pgconductor._private_executions parent + where parent.waiting_on_execution_id = any(v_ids) + and parent.completed_at is null and parent.failed_at is null and parent.locked_by is null + ); + exit when cardinality(v_ids) = 0; + v_ancestors := v_ancestors || v_ids; + end loop; + + -- ancestors are locked child first, then the chain is followed again on the locked rows + return query + with recursive locked_ancestors as materialized ( + select parent.id, parent.queue, parent.waiting_on_execution_id, parent.task_key, + parent."group", parent.payload, parent.attempts, parent.trace_context, parent.metadata, + parent.result_waiters, + t.remove_on_fail_days = 0 as should_remove, + t.dead_letter_queue, t.dead_letter_task_key + from pgconductor._private_executions parent + join pgconductor._private_tasks t on t.key = parent.task_key and t.queue = parent.queue + where parent.id = any(v_ancestors) + and parent.completed_at is null and parent.failed_at is null and parent.locked_by is null + order by array_position(v_ancestors, parent.id) + for update of parent + ), + targets as ( + select c.error, c.cancelled, a.* + from jsonb_to_recordset(p_children) as c(execution_id uuid, error text, cancelled boolean) + join locked_ancestors a on a.waiting_on_execution_id = c.execution_id + union all + select t.error, t.cancelled, a.* + from targets t + join locked_ancestors a on a.waiting_on_execution_id = t.id + ), + removed as ( + delete from pgconductor._private_executions e + using targets t + where e.id = any(v_ancestors) and e.id = t.id and e.queue = t.queue + and t.should_remove and e.locked_by is null + ), + failed as ( + update pgconductor._private_executions e + set failed_at = pgconductor._private_current_time(), + last_error = 'Child execution failed: ' || t.error, + waiting_on_execution_id = null, waiting_step_key = null, + locked_by = null, locked_at = null + from targets t + where e.id = any(v_ancestors) and e.id = t.id and e.queue = t.queue + and t.should_remove is not true + returning e.id, e.queue + ), + deleted_waits as ( + delete from pgconductor._private_custom_event_subscriptions subscription + using failed f + where subscription.kind = 'execution_wait' + and subscription.execution_id = f.id + and subscription.queue = f.queue + ) + select t.id, t.queue, t.task_key, t."group", t.payload, 'Child execution failed: ' || t.error, + t.attempts, t.cancelled, t.dead_letter_queue, t.dead_letter_task_key, t.trace_context, + t.metadata, t.result_waiters + from targets t; +end; +$function$; + -- cancel an execution create or replace function pgconductor.cancel_execution( p_execution_id uuid, diff --git a/packages/pgconductor-js/src/query-builder.ts b/packages/pgconductor-js/src/query-builder.ts index 27a1364..afa174c 100644 --- a/packages/pgconductor-js/src/query-builder.ts +++ b/packages/pgconductor-js/src/query-builder.ts @@ -606,6 +606,7 @@ export class QueryBuilder { buildReturnExecutions({ orchestratorId, + queueName, results, }: ReturnExecutionsArgs): PendingQuery | null { if (results.length === 0) return null; @@ -617,12 +618,19 @@ export class QueryBuilder { return r; }); + // Every reference to the executions table is narrowed to the worker's queue, so it is planned + // for one partition; parents may be in any queue, so functions look them up only when needed. + // The queue is a literal rather than a parameter, because a cached generic plan prunes only by + // constants: each queue gets its own prepared statement, planned once per connection. + const queue = this.sql.unsafe( + `E'${queueName.replaceAll("\\", "\\\\").replaceAll("'", "''")}'::text`, + ); const ctes: PendingQuery[] = []; ctes.push(this.sql`now_ts as (select pgconductor._private_current_time() as ts)`); ctes.push(this.sql`result_data as ( select * from jsonb_to_recordset(${this.sql.json(rows)}::jsonb) as r( - execution_id uuid, queue text, task_key text, claim_token uuid, status text, + execution_id uuid, task_key text, claim_token uuid, status text, result jsonb, error text, reschedule_in_ms bigint, step_key text, timeout_ms bigint, child_task_name text, child_task_queue text, child_payload jsonb, @@ -634,23 +642,26 @@ export class QueryBuilder { ctes.push(this.sql`valid_results as materialized ( select r.*, e.cancelled as execution_cancelled, e.last_error as execution_last_error, - e.subscription_id, e.signal_count as execution_signal_count, e.result_waiters + e.subscription_id, e.parent_execution_id, e.cancel_with_parent, + e.signal_count as execution_signal_count, e.result_waiters, + e."group" as execution_group, e.payload as execution_payload, + e.attempts as execution_attempts, e.trace_context as execution_trace_context, + e.metadata as execution_metadata from result_data r join pgconductor._private_executions e on e.id = r.execution_id - and e.queue = r.queue + and e.queue = ${queue} and e.task_key = r.task_key and e.locked_by = ${orchestratorId}::uuid and e.claim_token = r.claim_token for update of e )`); ctes.push(this.sql`task_configs as materialized ( - select t.queue, t.key, t.max_attempts, t.remove_on_complete_days, t.remove_on_fail_days, + select t.key, t.max_attempts, t.remove_on_complete_days, t.remove_on_fail_days, t.dead_letter_queue, t.dead_letter_task_key - from ( - select distinct queue, task_key from valid_results - ) r - join pgconductor._private_tasks t on t.queue = r.queue and t.key = r.task_key + from pgconductor._private_tasks t + where t.queue = ${queue} + and t.key in (select task_key from valid_results) )`); ctes.push(this.sql`completed_candidates as ( select * from valid_results where status = 'completed' and not execution_cancelled @@ -659,7 +670,7 @@ export class QueryBuilder { // locked row, so it counts signals committed after this statement's snapshot, which the // count below cannot see. Signals named by no listener of the attempt do not re-queue it. ctes.push(this.sql`signal_requeued as materialized ( - select c.execution_id, c.queue + select c.execution_id from completed_candidates c where c.execution_signal_count > 0 and ( @@ -689,7 +700,7 @@ export class QueryBuilder { set attempts = greatest(e.attempts - 1, 0), run_at = nt.ts, locked_by = null, locked_at = null from now_ts nt, signal_requeued q - where e.id = q.execution_id and e.queue = q.queue + where e.id = q.execution_id and e.queue = ${queue} and e.locked_by = ${orchestratorId}::uuid returning e.id )`); @@ -704,59 +715,37 @@ export class QueryBuilder { select * from valid_results where status = 'invoke_child' and not execution_cancelled )`); - // A completed child may wake only a parent which is still waiting and is not - // currently claimed. The row lock makes this check race-safe. - ctes.push(this.sql`completed_parents as materialized ( - select parent.id as parent_id, parent.queue, parent.waiting_step_key, r.result - from completed_results r - join pgconductor._private_executions parent - on parent.waiting_on_execution_id = r.execution_id - where r.subscription_id is null - and parent.completed_at is null - and parent.failed_at is null - and parent.locked_by is null - for update of parent - )`); - ctes.push(this.sql`parent_steps_all as ( - insert into pgconductor._private_steps (execution_id, queue, key, result) - select parent_id, queue, waiting_step_key, - jsonb_build_object('status', 'completed', 'result', result) - from completed_parents - where waiting_step_key is not null - on conflict (execution_id, key) do nothing - returning execution_id + // Wakes the parents waiting on completed children; a child no parent waits on any more fails. + // The parent functions read with fresh snapshots, so they run first in writes, before this + // statement changes any row they could read. + ctes.push(this.sql`orphan_ids as materialized ( + select o.id + from ( + select jsonb_agg(jsonb_build_object('execution_id', execution_id, 'result', result)) as children + from completed_results + where parent_execution_id is not null + and subscription_id is null + and not cancel_with_parent + having count(*) > 0 + ) c + cross join lateral pgconductor._private_complete_children(c.children) as o(id) )`); ctes.push(this.sql`orphaned_children as ( update pgconductor._private_executions e set failed_at = nt.ts, completed_at = null, last_error = 'Parent timed out before child completed', locked_by = null, locked_at = null - from now_ts nt, completed_results r - where e.id = r.execution_id and e.queue = r.queue + from now_ts nt, orphan_ids o + where e.id = o.id and e.queue = ${queue} and e.locked_by = ${orchestratorId}::uuid - and e.parent_execution_id is not null - and e.subscription_id is null - and not e.cancel_with_parent - and not exists ( - select 1 from pgconductor._private_executions parent - where parent.waiting_on_execution_id = r.execution_id - ) - returning e.id - )`); - ctes.push(this.sql`updated_parents_all as ( - update pgconductor._private_executions e - set run_at = nt.ts, waiting_on_execution_id = null, waiting_step_key = null, - locked_by = null, locked_at = null - from now_ts nt, completed_parents p - where e.id = p.parent_id and e.queue = p.queue returning e.id )`); ctes.push(this.sql`deleted_completed as ( delete from pgconductor._private_executions e using completed_results r, task_configs tc - where e.id = r.execution_id and e.queue = r.queue + where e.id = r.execution_id and e.queue = ${queue} and e.locked_by = ${orchestratorId}::uuid - and tc.key = r.task_key and tc.queue = r.queue and tc.remove_on_complete_days = 0 + and tc.key = r.task_key and tc.remove_on_complete_days = 0 and not exists (select 1 from orphaned_children oc where oc.id = e.id) returning e.id )`); @@ -764,93 +753,55 @@ export class QueryBuilder { update pgconductor._private_executions e set completed_at = nt.ts, result = r.result, locked_by = null, locked_at = null from now_ts nt, completed_results r, task_configs tc - where e.id = r.execution_id and e.queue = r.queue + where e.id = r.execution_id and e.queue = ${queue} and e.locked_by = ${orchestratorId}::uuid - and tc.key = r.task_key and tc.queue = r.queue + and tc.key = r.task_key and (tc.remove_on_complete_days is null or tc.remove_on_complete_days != 0) and not exists (select 1 from orphaned_children oc where oc.id = e.id) - returning e.id, e.queue + returning e.id )`); ctes.push(this.sql`permanently_failed_children as materialized ( - select r.execution_id, r.queue, r.task_key, - r.execution_cancelled, + select r.execution_id, r.task_key, r.execution_cancelled, -- the cancellation reason on the locked row, not the attempt's view of it coalesce(case when r.execution_cancelled then r.execution_last_error end, r.error, r.execution_last_error, 'unknown error') as child_error, - e."group" as execution_group, - e.payload as execution_payload, - e.attempts as execution_attempts, - e.subscription_id, - e.trace_context, - e.metadata, - r.result_waiters, + r.execution_group, r.execution_payload, r.execution_attempts, + r.subscription_id, r.parent_execution_id, + r.execution_trace_context, r.execution_metadata, r.result_waiters, tc.remove_on_fail_days = 0 as should_remove, tc.dead_letter_queue, tc.dead_letter_task_key from failed_results r - join pgconductor._private_executions e on e.id = r.execution_id and e.queue = r.queue - join task_configs tc on tc.key = r.task_key and tc.queue = r.queue - where e.attempts >= tc.max_attempts + join task_configs tc on tc.key = r.task_key + where r.execution_attempts >= tc.max_attempts or r.status = 'permanently_failed' or r.execution_cancelled )`); - // A terminal failure fails every workflow still waiting on it. Ancestors are - // locked child first, then the chain is followed again on the locked rows. - ctes.push(this.sql`waiting_ancestors as ( - select parent.id, 1 as depth - from permanently_failed_children p - join pgconductor._private_executions parent - on parent.waiting_on_execution_id = p.execution_id - where p.subscription_id is null - and parent.completed_at is null and parent.failed_at is null and parent.locked_by is null - union all - select parent.id, a.depth + 1 - from waiting_ancestors a - join pgconductor._private_executions parent - on parent.waiting_on_execution_id = a.id - where parent.completed_at is null and parent.failed_at is null and parent.locked_by is null - )`); - ctes.push(this.sql`locked_ancestors as materialized ( - select parent.id as parent_id, parent.queue as parent_queue, - parent.waiting_on_execution_id, - parent.task_key as parent_task_key, parent."group" as parent_group, - parent.payload as parent_payload, parent.attempts as parent_attempts, - parent.trace_context as parent_trace_context, - parent.metadata as parent_metadata, - parent.result_waiters as parent_result_waiters, - pt.remove_on_fail_days = 0 as parent_should_remove, - pt.dead_letter_queue as parent_dead_letter_queue, - pt.dead_letter_task_key as parent_dead_letter_task_key - from waiting_ancestors a - join pgconductor._private_executions parent on parent.id = a.id - join pgconductor._private_tasks pt - on pt.key = parent.task_key and pt.queue = parent.queue - where parent.completed_at is null and parent.failed_at is null and parent.locked_by is null - order by a.depth - for update of parent - )`); - ctes.push(this.sql`failed_parent_targets as ( - select p.child_error, p.execution_cancelled as child_cancelled, a.* - from permanently_failed_children p - join locked_ancestors a on a.waiting_on_execution_id = p.execution_id - where p.subscription_id is null - union all - select f.child_error, f.child_cancelled, a.* - from failed_parent_targets f - join locked_ancestors a on a.waiting_on_execution_id = f.parent_id + // A terminal failure fails every workflow still waiting on it + ctes.push(this.sql`failed_parents as materialized ( + select p.* + from ( + select jsonb_agg(jsonb_build_object( + 'execution_id', execution_id, 'error', child_error, 'cancelled', execution_cancelled + )) as children + from permanently_failed_children + where parent_execution_id is not null and subscription_id is null + having count(*) > 0 + ) c + cross join lateral pgconductor._private_fail_waiting_parents(c.children) p )`); ctes.push(this.sql`terminal_failures as materialized ( - select p.execution_id, p.queue, p.task_key, p.execution_group, p.execution_payload as payload, - p.child_error as failure_error, p.execution_attempts as failure_attempts, - p.execution_cancelled, p.dead_letter_queue, p.dead_letter_task_key, p.trace_context, - p.metadata + select p.execution_id, ${queue} as queue, p.task_key, p.execution_group, + p.execution_payload as payload, p.child_error as failure_error, + p.execution_attempts as failure_attempts, p.execution_cancelled, + p.dead_letter_queue, p.dead_letter_task_key, + p.execution_trace_context as trace_context, p.execution_metadata as metadata from permanently_failed_children p union all - select p.parent_id, p.parent_queue, p.parent_task_key, p.parent_group, p.parent_payload, - 'Child execution failed: ' || p.child_error, p.parent_attempts, p.child_cancelled, - p.parent_dead_letter_queue, p.parent_dead_letter_task_key, p.parent_trace_context, - p.parent_metadata - from failed_parent_targets p + select p.execution_id, p.queue, p.task_key, p.execution_group, p.payload, + p.failure_error, p.failure_attempts, p.execution_cancelled, + p.dead_letter_queue, p.dead_letter_task_key, p.trace_context, p.metadata + from failed_parents p )`); ctes.push(this.sql`dead_lettered as ( insert into pgconductor._private_executions (task_key, queue, payload, run_at, "group", trace_context, metadata, dead_letter) @@ -874,52 +825,34 @@ export class QueryBuilder { do nothing returning id )`); - ctes.push(this.sql`failed_updates as ( - select p.execution_id as target_id, p.queue, p.child_error, true as is_child - from permanently_failed_children p - where p.should_remove is not true - union all - select p.parent_id, p.parent_queue, p.child_error, false - from failed_parent_targets p - where p.parent_should_remove is not true - )`); - ctes.push(this.sql`failed_delete_targets as materialized ( - select p.execution_id as target_id, p.queue, ${orchestratorId}::uuid as expected_locked_by - from permanently_failed_children p - where p.should_remove is true - union all - select p.parent_id, p.parent_queue, null::uuid - from failed_parent_targets p - where p.parent_should_remove is true - )`); ctes.push(this.sql`deleted_failed as ( delete from pgconductor._private_executions e - using failed_delete_targets f - where e.id = f.target_id and e.queue = f.queue - and e.locked_by is not distinct from f.expected_locked_by + using permanently_failed_children p + where e.id = p.execution_id and e.queue = ${queue} + and e.locked_by = ${orchestratorId}::uuid + and p.should_remove is true returning e.id )`); ctes.push(this.sql`updated_failed as ( update pgconductor._private_executions e - set failed_at = nt.ts, - last_error = case when f.is_child then coalesce(f.child_error, 'unknown error') - else 'Child execution failed: ' || coalesce(f.child_error, 'unknown error') end, + set failed_at = nt.ts, last_error = p.child_error, waiting_on_execution_id = null, waiting_step_key = null, locked_by = null, locked_at = null - from now_ts nt, failed_updates f - where e.id = f.target_id and e.queue = f.queue - returning e.id, e.queue + from now_ts nt, permanently_failed_children p + where e.id = p.execution_id and e.queue = ${queue} + and p.should_remove is not true + returning e.id )`); ctes.push(this.sql`cleaned_terminal_event_waits as ( delete from pgconductor._private_custom_event_subscriptions subscription using ( - select id, queue from updated_completed + select id from updated_completed union all - select id, queue from updated_failed + select id from updated_failed ) terminal where subscription.kind = 'execution_wait' and subscription.execution_id = terminal.id - and subscription.queue = terminal.queue + and subscription.queue = ${queue} returning subscription.id )`); // Waits for a settled execution are read from its locked row, which also holds waits @@ -943,9 +876,8 @@ export class QueryBuilder { end from permanently_failed_children p union all - select p.parent_result_waiters, - jsonb_build_object('status', 'failed', 'error', 'Child execution failed: ' || p.child_error) - from failed_parent_targets p + select p.result_waiters, jsonb_build_object('status', 'failed', 'error', p.failure_error) + from failed_parents p ) s cross join lateral jsonb_array_elements(s.result_waiters) w having count(*) > 0 @@ -957,9 +889,9 @@ export class QueryBuilder { ((array[15, 30, 60, 120, 300, 600, 1200, 2400, 3600, 7200])[least(greatest(e.attempts, 1), 10)] * interval '1 second'), locked_by = null, locked_at = null from now_ts nt, failed_results r, task_configs tc - where e.id = r.execution_id and e.queue = r.queue + where e.id = r.execution_id and e.queue = ${queue} and e.locked_by = ${orchestratorId}::uuid - and tc.key = r.task_key and tc.queue = r.queue + and tc.key = r.task_key and r.status <> 'permanently_failed' and not r.execution_cancelled and e.attempts < tc.max_attempts @@ -968,7 +900,7 @@ export class QueryBuilder { ctes.push(this.sql`released_steps as ( insert into pgconductor._private_steps (execution_id, queue, key, result) - select execution_id, queue, step_key, null::jsonb from released_results + select execution_id, ${queue}, step_key, null::jsonb from released_results where step_key is not null on conflict (execution_id, key) do nothing returning execution_id @@ -980,7 +912,7 @@ export class QueryBuilder { else nt.ts + coalesce(r.reschedule_in_ms, 0) * interval '1 millisecond' end, locked_by = null, locked_at = null from now_ts nt, released_results r - where e.id = r.execution_id and e.queue = r.queue + where e.id = r.execution_id and e.queue = ${queue} and e.locked_by = ${orchestratorId}::uuid returning e.id )`); @@ -989,11 +921,6 @@ export class QueryBuilder { insert into pgconductor._private_executions (id, task_key, queue, payload, run_at, parent_execution_id, "group", trace_context, metadata) select pgconductor._private_portable_uuidv7(), r.child_task_name, r.child_task_queue, r.child_payload, nt.ts, r.execution_id, r."group", r.trace_context, r.metadata from invoke_child_data r, now_ts nt - where exists ( - select 1 from pgconductor._private_executions parent - where parent.id = r.execution_id and parent.queue = r.queue - and parent.locked_by = ${orchestratorId}::uuid - ) returning id, parent_execution_id )`); ctes.push(this.sql`updated_invoke_parents as ( @@ -1004,16 +931,15 @@ export class QueryBuilder { locked_by = null, locked_at = null from now_ts nt, inserted_children ic join invoke_child_data r on r.execution_id = ic.parent_execution_id - where e.id = r.execution_id and e.queue = r.queue + where e.id = r.execution_id and e.queue = ${queue} and e.locked_by = ${orchestratorId}::uuid returning e.id )`); ctes.push(this.sql`writes as ( - select 1 from updated_signal_requeued - union all select 1 from parent_steps_all - union all select 1 from orphaned_children - union all select 1 from updated_parents_all + select 1 from orphaned_children + union all select 1 from failed_parents + union all select 1 from updated_signal_requeued union all select 1 from deleted_completed union all select 1 from updated_completed union all select 1 from dead_lettered diff --git a/packages/pgconductor-js/src/worker.ts b/packages/pgconductor-js/src/worker.ts index 0ffa477..11c2e62 100644 --- a/packages/pgconductor-js/src/worker.ts +++ b/packages/pgconductor-js/src/worker.ts @@ -488,7 +488,6 @@ export class Worker< .map((exec) => ({ execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, status: "permanently_failed", error: exec.last_error || "Execution was cancelled", @@ -505,7 +504,6 @@ export class Worker< ...activeExecs.map((exec) => ({ execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, status: "released" as const, })), @@ -565,7 +563,6 @@ export class Worker< const identity = { execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: exec.task_key, }; const signals = new SignalInbox({ @@ -789,7 +786,6 @@ export class Worker< rejected.push({ execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, ...payload, }); @@ -880,7 +876,6 @@ export class Worker< return executions.map((exec) => ({ execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, status: "released" as const, reschedule_in_ms: result.reschedule_in_ms, @@ -893,7 +888,6 @@ export class Worker< return executions.map((exec) => ({ execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, status: "released" as const, })); @@ -903,7 +897,6 @@ export class Worker< return executions.map((exec) => ({ execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, status: "failed" as const, error: @@ -918,7 +911,6 @@ export class Worker< const identity = { execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, }; const result = await parseResult(task, outcome.output?.[i]); @@ -935,7 +927,6 @@ export class Worker< return executions.map((exec) => ({ execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, status: "released" as const, })); @@ -946,7 +937,6 @@ export class Worker< return executions.map((exec) => ({ execution_id: exec.id, claim_token: exec.claim_token, - queue: exec.queue, task_key: taskKey, status: "failed" as const, error, @@ -1011,7 +1001,11 @@ export class Worker< await this.telemetry.settle({ queue: this.queueName, batchMessageCount: results.length, - run: () => this.db.returnExecutions({ orchestratorId, results }, { signal }), + run: () => + this.db.returnExecutions( + { orchestratorId, queueName: this.queueName, results }, + { signal }, + ), }); } catch (err) { buffer = [...results, ...buffer]; diff --git a/packages/pgconductor-js/tests/fixtures/sessions.ts b/packages/pgconductor-js/tests/fixtures/sessions.ts index 2dcfb17..b6d9ba8 100644 --- a/packages/pgconductor-js/tests/fixtures/sessions.ts +++ b/packages/pgconductor-js/tests/fixtures/sessions.ts @@ -113,9 +113,13 @@ export async function openSession(db: TestDatabase): Promise { }; } -// The settlement statement a worker flushes, run on the given session +// The settlement statement a worker of the default queue flushes, run on the given session export function settle(sql: Sql, orchestratorId: string, results: ExecutionResult[]) { - return new QueryBuilder(sql).buildReturnExecutions({ orchestratorId, results }); + return new QueryBuilder(sql).buildReturnExecutions({ + orchestratorId, + queueName: "default", + results, + }); } // Barrier: resolves once `waiter` waits for a lock held by `holder` diff --git a/packages/pgconductor-js/tests/integration/cancellation-cascades.test.ts b/packages/pgconductor-js/tests/integration/cancellation-cascades.test.ts index dd1813b..9ae22d6 100644 --- a/packages/pgconductor-js/tests/integration/cancellation-cascades.test.ts +++ b/packages/pgconductor-js/tests/integration/cancellation-cascades.test.ts @@ -70,7 +70,6 @@ describe("cancellation cascades (Postgres integration)", () => { await settle(a.sql, ORCH_A, [ { execution_id: L, - queue: "default", task_key: "linked", claim_token: tokenL, status: "invoke_child", @@ -96,7 +95,6 @@ describe("cancellation cascades (Postgres integration)", () => { settle(b.sql, ORCH_B, [ { execution_id: P, - queue: "default", task_key: "parent", claim_token: tokenP, status: "permanently_failed", @@ -136,7 +134,6 @@ describe("cancellation cascades (Postgres integration)", () => { await settle(db.sql, ORCH_A, [ { execution_id: P, - queue: "default", task_key: "parent", claim_token: token, status: "permanently_failed", @@ -144,7 +141,6 @@ describe("cancellation cascades (Postgres integration)", () => { }, { execution_id: sleeping, - queue: "default", task_key: "linked", claim_token: token, status: "released", @@ -153,7 +149,6 @@ describe("cancellation cascades (Postgres integration)", () => { }, { execution_id: completing, - queue: "default", task_key: "linked", claim_token: token, status: "completed", diff --git a/packages/pgconductor-js/tests/integration/cancellation.test.ts b/packages/pgconductor-js/tests/integration/cancellation.test.ts index f11a477..a41a8d2 100644 --- a/packages/pgconductor-js/tests/integration/cancellation.test.ts +++ b/packages/pgconductor-js/tests/integration/cancellation.test.ts @@ -1121,9 +1121,10 @@ describe("Cancellation Support", () => { expect(await db.client.cancelExecution(sleeping)).toBe(true); expect(await db.client.cancelExecution(invoking)).toBe(true); - const execution = { queue: "default", task_key: "workflow-task", claim_token: claimToken }; + const execution = { task_key: "workflow-task", claim_token: claimToken }; await db.client.returnExecutions({ orchestratorId: orchestrator.id, + queueName: "default", results: [ { ...execution, execution_id: sleeping, status: "released", reschedule_in_ms: "infinity" }, { @@ -1182,11 +1183,11 @@ describe("Cancellation Support", () => { await db.client.returnExecutions({ orchestratorId: orchestrator.id, + queueName: "default", results: [ { execution_id: child, claim_token: claimToken, - queue: "default", task_key: "workflow-task", status: "permanently_failed", error: "boom", diff --git a/packages/pgconductor-js/tests/integration/dead-letter.test.ts b/packages/pgconductor-js/tests/integration/dead-letter.test.ts index a32682e..da1191a 100644 --- a/packages/pgconductor-js/tests/integration/dead-letter.test.ts +++ b/packages/pgconductor-js/tests/integration/dead-letter.test.ts @@ -189,17 +189,18 @@ describe("dead-letter queues (Postgres integration)", () => { const result = { execution_id: execution.id, claim_token: execution.claim_token, - queue: execution.queue, task_key: execution.task_key, status: "permanently_failed" as const, error: "settlement failure", }; await db.client.returnExecutions({ orchestratorId, + queueName: "default", results: [{ ...result }], }); await db.client.returnExecutions({ orchestratorId, + queueName: "default", results: [{ ...result }], }); const rows = await db.sql<{ count: string; source: string | null }[]>` @@ -211,6 +212,7 @@ describe("dead-letter queues (Postgres integration)", () => { // A result from a different worker is fenced out as well. await db.client.returnExecutions({ orchestratorId: crypto.randomUUID(), + queueName: "default", results: [{ ...result }], }); const count = await db.sql<{ count: string }[]>` @@ -262,11 +264,11 @@ describe("dead-letter queues (Postgres integration)", () => { await db.client.returnExecutions({ orchestratorId, + queueName: execution.queue, results: [ { execution_id: execution.id, claim_token: execution.claim_token, - queue: execution.queue, task_key: execution.task_key, status: "permanently_failed", error: "boom", @@ -368,11 +370,11 @@ describe("dead-letter queues (Postgres integration)", () => { expect(await db.client.cancelExecution(executionId)).toBe(true); await db.client.returnExecutions({ orchestratorId, + queueName: execution.queue, results: [ { execution_id: execution.id, claim_token: execution.claim_token, - queue: execution.queue, task_key: execution.task_key, status: "permanently_failed", error: "Task was cancelled", @@ -431,11 +433,11 @@ describe("dead-letter queues (Postgres integration)", () => { await db.client.returnExecutions({ orchestratorId: parent.locked_by, + queueName: parent.queue, results: [ { execution_id: parent.id, claim_token: parent.claim_token, - queue: parent.queue, task_key: parent.task_key, status: "invoke_child", timeout_ms: "infinity", @@ -460,11 +462,11 @@ describe("dead-letter queues (Postgres integration)", () => { expect(await db.client.cancelExecution(child.id)).toBe(true); await db.client.returnExecutions({ orchestratorId: childOrchestratorId, + queueName: child.queue, results: [ { execution_id: child.id, claim_token: child.claim_token, - queue: child.queue, task_key: child.task_key, status: "permanently_failed", error: "Task was cancelled", @@ -482,6 +484,116 @@ describe("dead-letter queues (Postgres integration)", () => { expect(counts).toEqual({ source: "0", destination: "0" }); }, 15000); + test("fails the waiting ancestors of a failed child and delivers them to their DLQ", async () => { + const db = await pool.child(); + databases.push(db); + const conductor = Conductor.create({ sql: db.sql, context: {} }); + await conductor.ensureInstalled(); + const queues = { + workflows: [{ key: "root", queue: "workflows" }], + default: [ + { + key: "step-parent", + queue: "default", + deadLetterQueue: "dlq", + deadLetterTaskKey: "step-parent-failed", + }, + ], + children: [{ key: "leaf", queue: "children", maxAttempts: 1 }], + }; + for (const [queueName, taskSpecs] of Object.entries(queues)) { + await db.client.registerWorker({ + queueName, + taskSpecs, + cronSchedules: [], + eventSubscriptions: [], + }); + } + const orchestratorId = await db.registerOrchestrator(); + const claim = async (queueName: string, taskKey: string) => { + const [execution] = ( + await db.client.getExecutions({ + orchestratorId, + queueName, + batchSize: 1, + taskKeys: [taskKey], + }) + ).executions; + if (!execution) throw new Error(`expected claimed ${taskKey}`); + return execution; + }; + const invokeChild = async ( + parent: { id: string; queue: string; task_key: string; claim_token: string }, + child: { key: string; queue: string }, + ) => { + await db.client.returnExecutions({ + orchestratorId, + queueName: parent.queue, + results: [ + { + execution_id: parent.id, + claim_token: parent.claim_token, + task_key: parent.task_key, + status: "invoke_child", + timeout_ms: "infinity", + step_key: "child", + child_task_name: child.key, + child_task_queue: child.queue, + child_payload: { from: parent.task_key }, + }, + ], + }); + }; + + await db.client.invoke({ task_key: "root", queue: "workflows" }); + const root = await claim("workflows", "root"); + await invokeChild(root, { key: "step-parent", queue: "default" }); + const parent = await claim("default", "step-parent"); + await invokeChild(parent, { key: "leaf", queue: "children" }); + const leaf = await claim("children", "leaf"); + await db.client.returnExecutions({ + orchestratorId, + queueName: "children", + results: [ + { + execution_id: leaf.id, + claim_token: leaf.claim_token, + task_key: leaf.task_key, + status: "permanently_failed", + error: "boom", + }, + ], + }); + + const failed = await db.sql<{ id: string; last_error: string | null; waiting: boolean }[]>` + select id, last_error, waiting_on_execution_id is not null as waiting + from pgconductor._private_executions + where id in (${root.id}::uuid, ${parent.id}::uuid, ${leaf.id}::uuid) and failed_at is not null + order by created_at + `; + expect(failed).toEqual([ + { id: root.id, last_error: "Child execution failed: boom", waiting: false }, + { id: parent.id, last_error: "Child execution failed: boom", waiting: false }, + { id: leaf.id, last_error: "boom", waiting: false }, + ]); + const delivered = await db.sql<{ task_key: string; payload: unknown; dead_letter: unknown }[]>` + select task_key, payload, dead_letter from pgconductor._private_executions where queue = 'dlq' + `; + expect(delivered).toEqual([ + { + task_key: "step-parent-failed", + payload: { from: "root" }, + dead_letter: expect.objectContaining({ + sourceExecutionId: parent.id, + sourceQueue: "default", + sourceTaskKey: "step-parent", + error: "Child execution failed: boom", + attempts: 1, + }), + }, + ]); + }, 15000); + test("rejects direct self-targets in the database but permits cross-queue identity", async () => { const db = await pool.child(); databases.push(db); @@ -577,7 +689,6 @@ describe("dead-letter queues (Postgres integration)", () => { const settlement = { execution_id: execution.id, claim_token: execution.claim_token, - queue: execution.queue, task_key: execution.task_key, status: "permanently_failed" as const, error: "rollback failure", @@ -585,6 +696,7 @@ describe("dead-letter queues (Postgres integration)", () => { await expect( db.client.returnExecutions({ orchestratorId: lockedBy, + queueName: "default", results: [settlement], }), ).rejects.toThrow("forced destination failure"); @@ -599,6 +711,7 @@ describe("dead-letter queues (Postgres integration)", () => { ); await db.client.returnExecutions({ orchestratorId: lockedBy, + queueName: "default", results: [settlement], }); const finalRows = await db.sql<{ source: string | null; source_failed: string }[]>` diff --git a/packages/pgconductor-js/tests/integration/event-pipeline.test.ts b/packages/pgconductor-js/tests/integration/event-pipeline.test.ts index a60bda8..479b22c 100644 --- a/packages/pgconductor-js/tests/integration/event-pipeline.test.ts +++ b/packages/pgconductor-js/tests/integration/event-pipeline.test.ts @@ -113,11 +113,11 @@ describe("event pipeline", () => { `; await db.client.returnExecutions({ orchestratorId, + queueName: INTERNAL_QUEUE, results: [ { execution_id: eventId, claim_token: source?.claim_token || "", - queue: INTERNAL_QUEUE, task_key: DISPATCH_TASK, status: "completed", }, @@ -1312,11 +1312,11 @@ describe("event pipeline", () => { if (!failedDestination) throw new Error("destination was not claimed"); await db.client.returnExecutions({ orchestratorId: destinationOwner, + queueName: failedDestination.queue, results: [ { execution_id: failedDestination.id, claim_token: failedDestination.claim_token, - queue: failedDestination.queue, task_key: failedDestination.task_key, status: "failed", error: "destination failed", @@ -1351,11 +1351,11 @@ describe("event pipeline", () => { if (!completedDestination) throw new Error("second destination was not claimed"); await db.client.returnExecutions({ orchestratorId: completionOwner, + queueName: completedDestination.queue, results: [ { execution_id: completedDestination.id, claim_token: completedDestination.claim_token, - queue: completedDestination.queue, task_key: completedDestination.task_key, status: "completed", }, diff --git a/packages/pgconductor-js/tests/integration/execution-foundations.test.ts b/packages/pgconductor-js/tests/integration/execution-foundations.test.ts index 12c65e1..a2044f4 100644 --- a/packages/pgconductor-js/tests/integration/execution-foundations.test.ts +++ b/packages/pgconductor-js/tests/integration/execution-foundations.test.ts @@ -33,11 +33,12 @@ describe("execution foundations", () => { } function grouped( - orchestratorId: string, + claimed: { locked_by: string; queue: string }, result: Parameters[0]["results"][number], ) { return { - orchestratorId, + orchestratorId: claimed.locked_by, + queueName: claimed.queue, results: [result], }; } @@ -111,19 +112,17 @@ describe("execution foundations", () => { if (!first || !second) throw new Error("expected both executions to be claimed"); await db.client.returnExecutions( - grouped(first.locked_by, { + grouped(first, { execution_id: first.id, claim_token: first.claim_token, - queue: first.queue, task_key: first.task_key, status: "completed", }), ); await db.client.returnExecutions( - grouped(second.locked_by, { + grouped(second, { execution_id: second.id, claim_token: second.claim_token, - queue: second.queue, task_key: second.task_key, status: "completed", }), @@ -322,11 +321,11 @@ describe("execution foundations", () => { await db.client.returnExecutions({ orchestratorId: parent.locked_by, + queueName: parent.queue, results: [ { execution_id: parent.id, claim_token: parent.claim_token, - queue: parent.queue, task_key: parent.task_key, status: "invoke_child", timeout_ms: 5000, @@ -357,11 +356,11 @@ describe("execution foundations", () => { await db.client.returnExecutions({ orchestratorId: child.locked_by, + queueName: child.queue, results: [ { execution_id: child.id, claim_token: child.claim_token, - queue: child.queue, task_key: child.task_key, status: "permanently_failed", error: "child failed", @@ -412,11 +411,11 @@ describe("execution foundations", () => { await db.client.returnExecutions({ orchestratorId: parent.locked_by, + queueName: parent.queue, results: [ { execution_id: parent.id, claim_token: parent.claim_token, - queue: parent.queue, task_key: parent.task_key, status: "invoke_child", timeout_ms: "infinity", @@ -439,11 +438,11 @@ describe("execution foundations", () => { if (!child) throw new Error("expected child claim"); await db.client.returnExecutions({ orchestratorId: child.locked_by, + queueName: child.queue, results: [ { execution_id: child.id, claim_token: child.claim_token, - queue: child.queue, task_key: child.task_key, status: "permanently_failed", error: "child failed", @@ -460,6 +459,100 @@ describe("execution foundations", () => { expect(outcome[0]?.last_error).toBe("Child execution failed: child failed"); }); + test("fails a completed child whose parent no longer waits on it", async () => { + const db = await database(); + await db.client.registerWorker({ + queueName: "parent-queue", + taskSpecs: [{ key: "parent", queue: "parent-queue" }], + cronSchedules: [], + eventSubscriptions: [], + }); + await db.client.registerWorker({ + queueName: "child-queue", + taskSpecs: [{ key: "child", queue: "child-queue" }], + cronSchedules: [], + eventSubscriptions: [], + }); + + const parentId = await db.client.invoke({ task_key: "parent", queue: "parent-queue" }); + if (!parentId) throw new Error("expected parent execution"); + const parent = ( + await db.client.getExecutions({ + orchestratorId: await db.registerOrchestrator(), + queueName: "parent-queue", + batchSize: 1, + taskKeys: ["parent"], + }) + ).executions[0]; + if (!parent) throw new Error("expected parent claim"); + await db.client.returnExecutions({ + orchestratorId: parent.locked_by, + queueName: parent.queue, + results: [ + { + execution_id: parent.id, + claim_token: parent.claim_token, + task_key: parent.task_key, + status: "invoke_child", + timeout_ms: "infinity", + step_key: "child-step", + child_task_name: "child", + child_task_queue: "child-queue", + child_payload: null, + }, + ], + }); + + const child = ( + await db.client.getExecutions({ + orchestratorId: await db.registerOrchestrator(), + queueName: "child-queue", + batchSize: 1, + taskKeys: ["child"], + }) + ).executions[0]; + if (!child) throw new Error("expected child claim"); + // the parent stopped waiting while the child ran + await db.sql` + update pgconductor._private_executions + set waiting_on_execution_id = null, waiting_step_key = null + where id = ${parentId}::uuid + `; + await db.client.returnExecutions({ + orchestratorId: child.locked_by, + queueName: child.queue, + results: [ + { + execution_id: child.id, + claim_token: child.claim_token, + task_key: child.task_key, + status: "completed", + result: { value: 1 }, + }, + ], + }); + + const rows = await db.sql< + { id: string; completed_at: Date | null; failed_at: Date | null; last_error: string | null }[] + >` + select id, completed_at, failed_at, last_error + from pgconductor._private_executions + where id in (${parentId}::uuid, ${child.id}::uuid) + `; + expect(rows.find((row) => row.id === child.id)).toMatchObject({ + completed_at: null, + failed_at: expect.any(Date), + last_error: "Parent timed out before child completed", + }); + expect(rows.find((row) => row.id === parentId)).toMatchObject({ + completed_at: null, + failed_at: null, + }); + expect( + await db.sql`select 1 from pgconductor._private_steps where execution_id = ${parentId}::uuid`, + ).toHaveLength(0); + }); + test("cancellation fences a buffered completion and does not retry it", async () => { const db = await database(); await db.client.registerWorker({ @@ -482,10 +575,9 @@ describe("execution foundations", () => { if (!claimed) throw new Error("expected claim"); await db.client.cancelExecution(executionId, { reason: "cancelled before flush" }); await db.client.returnExecutions( - grouped(claimed.locked_by, { + grouped(claimed, { execution_id: claimed.id, claim_token: claimed.claim_token, - queue: claimed.queue, task_key: claimed.task_key, status: "completed", }), @@ -539,11 +631,11 @@ describe("execution foundations", () => { const first = await claim(); const release = { orchestratorId, + queueName: "replayed", results: [ { execution_id: first.id, claim_token: first.claim_token, - queue: first.queue, task_key: first.task_key, status: "released" as const, reschedule_in_ms: 0, @@ -591,10 +683,9 @@ describe("execution foundations", () => { await blockerSql`begin`; await blockerSql`select 1 from pgconductor._private_executions where id = ${claimed.id}::uuid for update`; const settled = settler.returnExecutions( - grouped(claimed.locked_by, { + grouped(claimed, { execution_id: claimed.id, claim_token: claimed.claim_token, - queue: claimed.queue, task_key: claimed.task_key, status: "completed", }), @@ -657,11 +748,11 @@ describe("execution foundations", () => { if (!parent) throw new Error("expected parent claim"); await db.client.returnExecutions({ orchestratorId: parent.locked_by, + queueName: parent.queue, results: [ { execution_id: parent.id, claim_token: parent.claim_token, - queue: parent.queue, task_key: parent.task_key, status: "invoke_child", timeout_ms: "infinity", @@ -796,11 +887,11 @@ describe("execution foundations", () => { const staleBase = { execution_id: executionId, claim_token: oldClaim.claim_token, - queue: "fenced", task_key: "fenced-task", }; await db.client.returnExecutions({ orchestratorId: oldOrchestrator, + queueName: "fenced", results: [ { ...staleBase, status: "completed" }, { ...staleBase, status: "failed", error: "stale" }, @@ -824,10 +915,9 @@ describe("execution foundations", () => { expect(untouched[0]?.locked_by).toBe(newOrchestrator); await db.client.returnExecutions( - grouped(currentClaim.locked_by, { + grouped(currentClaim, { execution_id: currentClaim.id, claim_token: currentClaim.claim_token, - queue: currentClaim.queue, task_key: currentClaim.task_key, status: "completed", }), diff --git a/packages/pgconductor-js/tests/integration/lock-order.test.ts b/packages/pgconductor-js/tests/integration/lock-order.test.ts index 2b8bdaa..21a98b4 100644 --- a/packages/pgconductor-js/tests/integration/lock-order.test.ts +++ b/packages/pgconductor-js/tests/integration/lock-order.test.ts @@ -113,7 +113,6 @@ describe("lock order (Postgres integration)", () => { settle(b.sql, ORCH_B, [ { execution_id: C, - queue: "default", task_key: "child", claim_token: tokenC, status: "completed", @@ -133,7 +132,6 @@ describe("lock order (Postgres integration)", () => { settle(b.sql, ORCH_B, [ { execution_id: C, - queue: "default", task_key: "child", claim_token: tokenC, status: "permanently_failed", @@ -171,7 +169,6 @@ describe("lock order (Postgres integration)", () => { settle(b.sql, ORCH_B, [ { execution_id: C, - queue: "default", task_key: "linked", claim_token: tokenC, status: "completed", @@ -222,7 +219,6 @@ describe("lock order (Postgres integration)", () => { settle(b.sql, ORCH_B, [ { execution_id: Y, - queue: "default", task_key: "child", claim_token: tokenY, status: "completed", diff --git a/packages/pgconductor-js/tests/integration/queue-partitions.test.ts b/packages/pgconductor-js/tests/integration/queue-partitions.test.ts index c55c8ea..36e7c0e 100644 --- a/packages/pgconductor-js/tests/integration/queue-partitions.test.ts +++ b/packages/pgconductor-js/tests/integration/queue-partitions.test.ts @@ -70,6 +70,20 @@ describe("queue partitions", () => { await executeOnEachQueue(db, [`${prefix}-first`, `${prefix}-second`]); }, 30_000); + // settlement inlines the queue name as a literal + test("settles executions of queues whose names need quoting", async () => { + const db = await pool.child(); + databases.push(db); + const queues = ["it's", "back\\'slash"]; + + await executeOnEachQueue(db, queues); + + const rows = await db.sql<{ queue: string }[]>` + select queue from pgconductor._private_executions where completed_at is not null + `; + expect(rows.map((row) => row.queue).toSorted()).toEqual(queues.toSorted()); + }, 30_000); + test("drop_queue drops only the partition of the given queue", async () => { const db = await pool.child(); databases.push(db); diff --git a/packages/pgconductor-js/tests/integration/signals.test.ts b/packages/pgconductor-js/tests/integration/signals.test.ts index 069ae42..6241f7e 100644 --- a/packages/pgconductor-js/tests/integration/signals.test.ts +++ b/packages/pgconductor-js/tests/integration/signals.test.ts @@ -639,11 +639,11 @@ describe("Signals", () => { const first = await claim(); await db.client.returnExecutions({ orchestratorId, + queueName: "default", results: [ { execution_id: id, claim_token: first.claim_token, - queue: "default", task_key: "session", status: "released", reschedule_in_ms: 0, diff --git a/packages/pgconductor-js/tests/integration/wait-for-event.test.ts b/packages/pgconductor-js/tests/integration/wait-for-event.test.ts index 653a144..7557708 100644 --- a/packages/pgconductor-js/tests/integration/wait-for-event.test.ts +++ b/packages/pgconductor-js/tests/integration/wait-for-event.test.ts @@ -589,11 +589,11 @@ describe("waitForEvent", () => { const result = { execution_id: executionId, claim_token: execution.claim_token, - queue: "default", task_key: "wait.task", }; await database.client.returnExecutions({ orchestratorId, + queueName: "default", results: [ status === "completed" ? { ...result, status } : { ...result, status, error: "failed" }, ], @@ -1065,11 +1065,11 @@ describe("waitForEvent", () => { if (!second) throw new Error("expected claim"); await database.client.returnExecutions({ orchestratorId, + queueName: "default", results: [ { execution_id: executionId, claim_token: second.claim_token, - queue: "default", task_key: "wait.task", status: "failed", error: "failed before waiting again", diff --git a/packages/pgconductor-js/tests/integration/wait-for-result.test.ts b/packages/pgconductor-js/tests/integration/wait-for-result.test.ts index f16d228..6df3c3c 100644 --- a/packages/pgconductor-js/tests/integration/wait-for-result.test.ts +++ b/packages/pgconductor-js/tests/integration/wait-for-result.test.ts @@ -296,10 +296,10 @@ describe("ctx.waitForResult and execution branches (Postgres integration)", () = await registered.promise; const settling = database.client.returnExecutions({ orchestratorId, + queueName: "default", results: [ { execution_id: workId, - queue: "default", task_key: "work", claim_token: claimToken(workId), status: "completed", @@ -361,10 +361,10 @@ describe("ctx.waitForResult and execution branches (Postgres integration)", () = await database.client.returnExecutions({ orchestratorId, + queueName: "default", results: [ { execution_id: waiterId, - queue: "default", task_key: "work", claim_token: claimToken(waiterId), status: "completed", @@ -372,7 +372,6 @@ describe("ctx.waitForResult and execution branches (Postgres integration)", () = }, { execution_id: awaitedId, - queue: "default", task_key: "waiter", claim_token: claimToken(awaitedId), status: "completed",