[psteinroe]

conductor.getExecution() could not tell a sleeping or suspended execution from one waiting for its first attempt: all of them were pending. Operators also had no cheap way to see the depth of each queue.

Waiting status

getExecution (and the polling in conductor.waitForResult) reports a new waiting status with a waitingFor reason. It is derived from existing columns:

status: "pending" | "running" | "waiting" | "completed" | "failed" | "cancelled";
waitingFor: {
  type: "sleep" | "child" | "wait";  // wait = event, result or signal (ctx.waitForEvent/waitForResult/waitForAny)
  stepKey: string;
  executionId: string | null;        // the child of a "child" wait
  until: Date | null;                // wake-up time or deadline, null without one
} | null;

• child: waiting_on_execution_id is set.
• wait: waiting_step_key is set and there is no child.
• Every kind of wait counts only while run_at is still in the future. Once a deadline passes, the execution is pending until it is claimed.
• Retries after a backoff, delayed starts and time-window releases stay pending.

Sleep detection

A sleep step used to store null, which made a sleep in progress look the same as a retry that failed right after the sleep woke up. Settlement now stores the sleep's wake-up time in the same step insert ({until}) instead of null. An execution counts as sleeping only while its run_at equals that value, so a retry, window release or re-invoke that moves run_at ends the sleep. There are no new writes or columns; the existing insert just stores a different value.

pgconductor.queue_stats

A view with one row per queue: ready, oldest_ready_age, running, suspended (sleeping, waiting, backing off or delayed) and failed_last_hour (excluding cancellations). It is documented on a new Operations › Monitoring page.

Each count reads one existing partial index (is_available, active concurrency, failed_at), so the cost grows with unfinished executions and the last hour's failures, not with history. Two planner details matter:

• The time is read once in a materialized CTE. Otherwise the volatile _private_current_time() runs once per row inside the aggregate filters.
• failed_at is compared against a scalar subquery and a two-sided range. That keeps it an index condition with a good row estimate, so the plan stays under jit_above_cost. A per-queue lateral version cost about 45 ms of JIT compilation per call.

EXPLAIN on Postgres 15 with 995k executions (900k completed, 45k failed with 5k in the last hour, 40k ready, 9k suspended, 1k running) takes about 25 ms warm:

EXPLAIN (ANALYZE, BUFFERS)

 Nested Loop  (cost=35706.76..35706.92 rows=3 width=59) (actual time=28.621..28.643 rows=3 loops=1)
   Buffers: shared hit=1820
   CTE clock
     ->  Result  (cost=0.00..0.26 rows=1 width=8) (actual time=0.030..0.031 rows=1 loops=1)
   ->  CTE Scan on clock  (cost=0.00..0.02 rows=1 width=8) (actual time=0.032..0.033 rows=1 loops=1)
   ->  Hash Left Join  (cost=35706.50..35706.60 rows=3 width=51) (actual time=28.586..28.605 rows=3 loops=1)
         Hash Cond: (q.name = failed.queue)
         Buffers: shared hit=1820
         ->  Merge Left Join  (cost=23902.78..23902.87 rows=3 width=43) (actual time=26.693..26.708 rows=3 loops=1)
               Merge Cond: (q.name = running.queue)
               Buffers: shared hit=1681
               ->  Merge Left Join  (cost=23697.60..23697.64 rows=3 width=35) (actual time=26.361..26.373 rows=3 loops=1)
                     Merge Cond: (q.name = available.queue)
                     Buffers: shared hit=1652
                     ->  Sort  (cost=1.05..1.06 rows=3 width=11) (actual time=0.011..0.013 rows=3 loops=1)
                           Sort Key: q.name
                           Sort Method: quicksort  Memory: 25kB
                           Buffers: shared hit=1
                           ->  Seq Scan on _private_queues q  (cost=0.00..1.03 rows=3 width=11) (actual time=0.003..0.004 rows=3 loops=1)
                                 Buffers: shared hit=1
                     ->  Sort  (cost=23696.54..23696.55 rows=2 width=32) (actual time=26.344..26.348 rows=1 loops=1)
                           Sort Key: available.queue
                           Sort Method: quicksort  Memory: 25kB
                           Buffers: shared hit=1651
                           ->  Subquery Scan on available  (cost=23696.49..23696.53 rows=2 width=32) (actual time=26.334..26.338 rows=1 loops=1)
                                 Buffers: shared hit=1651
                                 ->  HashAggregate  (cost=23696.49..23696.51 rows=2 width=32) (actual time=26.334..26.337 rows=1 loops=1)
                                       Group Key: e.queue
                                       Batches: 1  Memory Usage: 24kB
                                       Buffers: shared hit=1651
                                       ->  Nested Loop  (cost=3090.21..22823.80 rows=49868 width=24) (actual time=2.782..18.168 rows=49000 loops=1)
                                             Buffers: shared hit=1651
                                             ->  CTE Scan on clock clock_1  (cost=0.00..0.02 rows=1 width=8) (actual time=0.000..0.002 rows=1 loops=1)
                                             ->  Append  (cost=3090.21..22325.10 rows=49868 width=16) (actual time=2.779..13.876 rows=49000 loops=1)
                                                   Buffers: shared hit=1651
                                                   ->  Bitmap Heap Scan on executions_default_c21f969b e_1  (cost=3090.21..22069.87 rows=49866 width=16) (actual time=2.779..10.995 rows=49000 loops=1)
                                                         Recheck Cond: is_available
                                                         Heap Blocks: exact=947
                                                         Buffers: shared hit=1650
                                                         ->  Bitmap Index Scan on idx_executions_default_c21f969b_get_executions  (cost=0.00..3077.74 rows=49866 width=0) (actual time=2.684..2.684 rows=49000 loops=1)
                                                               Buffers: shared hit=703
                                                   ->  Index Scan using idx_executions_other_795f3202_get_task_executions on executions_other_795f3202 e_2  (cost=0.12..5.89 rows=1 width=14) (actual time=0.013..0.013 rows=0 loops=1)
                                                         Buffers: shared hit=1
                                                   ->  Seq Scan on executions_pgconductor_int_08f24aee e_3  (cost=0.00..0.00 rows=1 width=40) (actual time=0.007..0.007 rows=0 loops=1)
                                                         Filter: is_available
               ->  Sort  (cost=205.19..205.19 rows=2 width=16) (actual time=0.329..0.331 rows=1 loops=1)
                     Sort Key: running.queue
                     Sort Method: quicksort  Memory: 25kB
                     Buffers: shared hit=29
                     ->  Subquery Scan on running  (cost=205.14..205.18 rows=2 width=16) (actual time=0.319..0.321 rows=1 loops=1)
                           Buffers: shared hit=29
                           ->  HashAggregate  (cost=205.14..205.16 rows=2 width=16) (actual time=0.318..0.319 rows=1 loops=1)
                                 Group Key: e_4.queue
                                 Batches: 1  Memory Usage: 24kB
                                 Buffers: shared hit=29
                                 ->  Append  (cost=0.15..204.59 rows=110 width=8) (actual time=0.021..0.204 rows=1000 loops=1)
                                       Buffers: shared hit=29
                                       ->  Index Scan using idx_executions_default_c21f969b_active_concurrency on executions_default_c21f969b e_5  (cost=0.15..198.15 rows=108 width=8) (actual time=0.020..0.135 rows=1000 loops=1)
                                             Buffers: shared hit=28
                                       ->  Index Scan using idx_executions_other_795f3202_active_concurrency on executions_other_795f3202 e_6  (cost=0.12..5.89 rows=1 width=6) (actual time=0.013..0.014 rows=0 loops=1)
                                             Buffers: shared hit=1
                                       ->  Seq Scan on executions_pgconductor_int_08f24aee e_7  (cost=0.00..0.00 rows=1 width=32) (actual time=0.001..0.001 rows=0 loops=1)
                                             Filter: ((locked_at IS NOT NULL) AND (failed_at IS NULL) AND (completed_at IS NULL))
         ->  Hash  (cost=11803.69..11803.69 rows=2 width=16) (actual time=1.881..1.884 rows=1 loops=1)
               Buckets: 1024  Batches: 1  Memory Usage: 9kB
               Buffers: shared hit=139
               ->  Subquery Scan on failed  (cost=11803.65..11803.69 rows=2 width=16) (actual time=1.874..1.877 rows=1 loops=1)
                     Buffers: shared hit=139
                     ->  HashAggregate  (cost=11803.65..11803.67 rows=2 width=16) (actual time=1.873..1.876 rows=1 loops=1)
                           Group Key: e_8.queue
                           Batches: 1  Memory Usage: 24kB
                           Buffers: shared hit=139
                           InitPlan 2 (returns $1)
                             ->  CTE Scan on clock clock_2  (cost=0.00..0.02 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=1)
                           InitPlan 3 (returns $2)
                             ->  CTE Scan on clock clock_3  (cost=0.00..0.02 rows=1 width=8) (actual time=0.000..0.001 rows=1 loops=1)
                           ->  Append  (cost=142.67..11778.82 rows=4957 width=8) (actual time=0.197..1.186 rows=4500 loops=1)
                                 Buffers: shared hit=139
                                 ->  Bitmap Heap Scan on executions_default_c21f969b e_9  (cost=142.67..10699.98 rows=4506 width=8) (actual time=0.197..0.916 rows=4500 loops=1)
                                       Recheck Cond: ((failed_at >= ($1 - '01:00:00'::interval)) AND (failed_at <= $2))
                                       Filter: (NOT cancelled)
                                       Rows Removed by Filter: 500
                                       Heap Blocks: exact=110
                                       Buffers: shared hit=137
                                       ->  Bitmap Index Scan on idx_executions_default_c21f969b_failed_cleanup  (cost=0.00..141.54 rows=4525 width=0) (actual time=0.182..0.182 rows=5000 loops=1)
                                             Index Cond: ((failed_at >= ($1 - '01:00:00'::interval)) AND (failed_at <= $2))
                                             Buffers: shared hit=27
                                 ->  Bitmap Heap Scan on executions_other_795f3202 e_10  (cost=4.25..1054.06 rows=450 width=6) (actual time=0.007..0.007 rows=0 loops=1)
                                       Recheck Cond: ((failed_at >= ($1 - '01:00:00'::interval)) AND (failed_at <= $2))
                                       Filter: (NOT cancelled)
                                       Buffers: shared hit=2
                                       ->  Bitmap Index Scan on idx_executions_other_795f3202_failed_cleanup  (cost=0.00..4.14 rows=450 width=0) (actual time=0.006..0.006 rows=0 loops=1)
                                             Index Cond: ((failed_at >= ($1 - '01:00:00'::interval)) AND (failed_at <= $2))
                                             Buffers: shared hit=2
                                 ->  Seq Scan on executions_pgconductor_int_08f24aee e_11  (cost=0.00..0.00 rows=1 width=32) (actual time=0.003..0.003 rows=0 loops=1)
                                       Filter: ((NOT cancelled) AND (failed_at <= $2) AND (failed_at >= ($1 - '01:00:00'::interval)))
 Planning:
   Buffers: shared hit=105
 Planning Time: 0.942 ms
 Execution Time: 28.810 ms
(100 rows)

Time: 30.452 ms

Verification

• New integration tests in execution-waiting-status.test.ts cover sleep, child with and without a timeout, event, signal, the return to pending after a sleep elapses, retries (including a failure right after a sleep), delayed starts, and waitForResult on a waiting execution. All 7 failed against main.
• queue-stats.test.ts covers the view under a fake clock.
• just lint, just format and pnpm typecheck pass. In the full pnpm test run, 622/623 tests passed. The one failure was wait-deadlines.test.ts › "waitForAny with an event and a completion settled after the deadline › times out", which passed 3/3 when run alone. It looks like a timing flake under the full parallel load: fake time keeps running and its deadline is 1 s.

Follow-ups

• If that wait-deadlines test fails again in CI, it needs a wider deadline margin.