Skip to content

perf(settlement): plan settlement for one partition - #103

Open
psteinroe wants to merge 1 commit into
mainfrom
perf/settlement-pruning
Open

psteinroe wants to merge 1 commit into
mainfrom
perf/settlement-pruning

Conversation

@psteinroe

Copy link
Copy Markdown
Owner

Settlement now plans for one partition, so its cost no longer grows with the number of queues (finding P10). Before, buildReturnExecutions referenced the partitioned executions table about 25 times and none of those references could be pruned. At 106 queues that cost ~520 ms of planning for each of the first 6 calls on every connection and ~30 ms per call after that. With Postgres' default jit = on, the statement's estimated cost also crossed jit_above_cost, so every settle JIT-compiled ~10.7k functions and took ~14 s.

One queue per statement

returnExecutions now takes the worker's queueName, and results no longer carry a queue each. Every reference to the executions table in the statement is narrowed to that queue. The queue is inlined as an escaped E'' literal rather than bound as a parameter, because a cached generic plan prunes partitions only by constants. So each queue gets its own prepared statement, planned a few times per connection and then served from the plan cache. With a parameter, Postgres kept choosing custom plans: ~3.5 ms of planning on every call, plus one ~250 ms generic plan per connection at 106 queues. A test covers queue names with a quote and a backslash, and the escaping round-trips under both standard_conforming_strings settings.

Parent paths move into functions

Waiting parents can be in any queue, so their lookups now run in two plpgsql functions that settlement calls only when a settled execution has a parent:

  • _private_complete_children() stores results for waiting parents, wakes them, and returns orphaned children (children no parent waits on any more).
  • _private_fail_waiting_parents() fails the chain of waiting ancestors and returns them for dead-lettering, result waits and the cancel cascade.

Their statements can't be pruned, so they use plan_cache_mode = force_generic_plan (planned once per connection) and look rows up by arrays of ids to stay on indexes: jsonb_to_recordset is always estimated at 100 rows, which otherwise led to a seq scan of every partition. Both functions read with fresh snapshots, so they run first in writes, before the statement changes any row they read. That keeps their view identical to the old single snapshot. The old existence check on the parent in inserted_children is dropped because valid_results already holds that row locked. Behaviour is unchanged otherwise: the full suite passes unchanged apart from the call shape. Two new tests cover the moved paths with parents in other queues: an orphaned child, and a three-level failure that dead-letters the middle ancestor. Both new tests also pass on main.

Measurements

The harness seeds ~1M executions: 93% completed, 3% failed, 3% pending, 1% suspended parents. It runs Postgres 15 with pg_stat_statements.track_planning. Each settle is one completion plus one retry, sent through DatabaseClient (postgres.js prepared statements). Times below are DB-side from pg_stat_statements; wall times on this machine were noisy.

before after
6 queues, first 6 calls per connection 18–29 ms planning each 3.5–4.5 ms planning each
6 queues, steady state 1.0–1.1 ms exec (cached generic plan) 0.5–0.6 ms exec, no planning
106 queues, first 6 calls per connection ~520 ms planning each (~3.1 s per connection) 3.5–7.7 ms planning each
106 queues, steady state 5.7 ms exec, ~30 ms wall (a generic plan locks every partition) 0.5–0.7 ms exec, no planning
106 queues, jit = on ~14 s per settle 0.8 ms exec

When every settled child wakes a parent in another queue (an invokeAndWait scenario), median wall time per settle drops from 39 ms to 9 ms at 106 queues and from 3.7 ms to 2.8 ms at 6 queues. At 106 queues the first call per connection takes ~170 ms (the functions' generic plans), against ~600–900 ms for each of the first 3 calls before.

Trade-off: each settled queue keeps its own cached plan of about 1 MB per connection. Before, there was one plan per connection, but it grew with the partition count: 2 MB at 6 queues, 16 MB at 106.

Follow-ups (out of scope)

  • Foreign-key checks into executions from steps and signals are still planned against every partition (P12, F11).
  • Claim planning still grows with queue count (P11, F12). The claim's estimated cost should be checked against jit_above_cost as well.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant