hotblocks: one heavy query holds the whole query pool, and new tasks queue behind it #118

Activity

1. mo4islona commented on Oct 16, 2026

   

   mo4islona

   on Oct 16, 2026

   ContributorAuthor

   More actions

   Update: offline reproduction, a fix behind a flag, and what makes a task heavy

   Offline reproduction

   A synthetic bench reproduces the problem and compares scheduling options. It models the query pool only:

   • Tasks arrive from a thread outside the pool, as POOL.spawn from tokio does.
   • Each task splits its work like the legacy engine: par_iter over scans, par_iter over columns, a rayon::join tree over pages, with a sequential part around it.
   • Light tasks follow the production run-time distribution. Heavy tasks are rare (about 1 in 1,000) and need 1–4 s of CPU each.

   The load is set so that the queue wait of an 8-thread pool matches production, with and without heavy tasks:

   queue wait, ms	mean	p50	p99
   bench, no heavy tasks	3.0	0.04	54
   production, normal periods	2.8	0.07	71
   bench, with heavy tasks	13–28	0.04	280–390
   production, slow periods	30	0.08	690

   A second effect: nested tasks. The queue wait is not the whole cost. A worker that waits in join runs other jobs (wait_until_cold → find_work in rayon-core), and these include whole query tasks from the injector, heavy ones too. The waiting task ends only after the task it picked up. In the bench this makes the light-task latency (spawn to end) 3–4 times its queue wait. #117 shows this effect in production as nested_ms.

   Results with heavy tasks, 120 s per run, the same seed for every policy. Times in ms; "own" is the run time without nested tasks.

   policy	light queue mean / p99 / p99.9	light latency mean / p99 / p99.9	heavy own, mean	heavy latency mean / max
   one shared pool of 8 (today)	20.2 / 337 / 505	60.7 / 1322 / 3726	599	1264 / 5070
   2 groups of 4, least loaded	9.1 / 273 / 747	20.6 / 445 / 1589	779	903 / 1898
   8 groups of 1, least loaded	4.2 / 38 / 1069	14.3 / 300 / 1322	2089	2091 / 3986
   thread per task, 8 workers	0.9 / 33 / 99	11.3 / 187 / 608	2167	2167 / 4115
   thread per task, 1/3 of the parallel work in polars	2.0 / 66 / 142	11.3 / 178 / 489	1593	1593 / 3007
   thread per task, 2/3 of the parallel work in polars	8.3 / 196 / 466	18.5 / 305 / 574	1061	1063 / 2175

   With no heavy tasks, the shared pool gives a light latency of 13.9 / 249 / 1175 ms and thread per task gives 10.2 / 184 / 602 ms.

   • Thread groups (option A) lower the means but not the tail. The least-loaded choice counts tasks, not their weight, so a light task can go behind a heavy one in the same group.
   • Thread per task: N rayon pools of one thread each take tasks from one FIFO queue. par_iter and join inside a task then run on the task's own thread, with no change to the engine code. Light tasks get the same latency as with no heavy tasks at all. Heavy tasks need 3.6× more own time, but their mean latency grows only 1.7× and their max falls, because in the shared pool they also wait for the tasks nested in them.
   • Polars keeps its global POOL, so the polars part of a query stays parallel and shared. With a third of the heavy parallel work in polars, light tasks keep almost all of the gain. With two thirds, part of the tail comes back, but it stays far below today.
   • The new engine uses the pool of its caller and has no static pool, so the same change covers it (Concurrent queries: some wait in the shared rayon pool while others keep finishing sqd-query-engine#57).

   Limits: the shape of the heavy tasks is a guess; a step that becomes several times slower can reach a client's read timeout; each run has only 25–38 heavy tasks, and a second seed for the shared pool gave 33.5 / 712 / 1649 ms, so the tails carry noise of about 2×.

   The fix, behind a flag

   --query-thread-per-task, also switchable at runtime with PATCH /settings {"query": {"threadPerTask": true}}, off by default. The gauge hotblocks_query_thread_per_task shows the mode of each replica. A PR follows. The rollout: switch one replica, watch the tail of hotblocks_query_task_run_seconds and client retries, then switch the others.

   What makes one task heavy

   The number of sub-tasks is not the problem: every query splits the same way, light ones into dozens of jobs too. The problem is the total work in a parallel phase. While it has jobs left, free workers take them before any new task.

   From production metrics:

   • A step stops near its time budget, but checks it only between chunks. So a task over 0.8 s is one chunk that took over 0.8 s to read. In slow periods such tasks are about 20 times more frequent.
   • Streams, chunks, blocks and bytes per dataset and client stay flat between slow and normal periods. So the cause is a small number of heavy queries, not more traffic, and the per-client volume metrics cannot show them.

   From the code, the cost of one chunk:

   • Scans ignore the requested block range. execute_scans reads the predicate columns for every row of the table in the chunk. The fromBlock..toBlock range is applied only in execute_output. After compaction a chunk holds up to 200,000 rows per table, so a query that needs one block of a large chunk pays for the whole chunk.
   • Relations (for example transaction → logs, instruction → inner instructions) run over every matched row, partly in polars.
   • Output reads every requested field for the matched rows, up to about 20 MB per chunk, and then encodes them to JSON.
   • The largest share of the slow-period cost came from the dataset kind with the widest and longest tables.

   Two hypotheses, not verified yet:

   1. A wide query (many fields and relations) that matches many rows.
   2. A stream behind the head whose fromBlock falls into large compacted chunks, so that every step scans whole chunks.

   If (2) holds, a separate and cheap fix is to apply the block range in the scans. If the rows of a table in a chunk are sorted by block number, the range is one contiguous run of rows. This needs a check of how the storage reader applies predicates.

   #117 names the queries: it logs every task that holds a worker for more than --slow-task-threshold-ms, with the query, the chunks and blocks it read, and the bytes it buffered.

   Next

   1. Open the thread-per-task PR and switch one replica.
   2. Deploy feat(hotblocks): count and log query tasks that hold a worker for long #117, then group its log lines by query shape and check the two hypotheses.
   3. Measure the polars share and the cost of one thread on the real heavy queries.
2. mo4islona commented on Oct 16, 2026

   

   mo4islona

   on Oct 16, 2026

   ContributorAuthor

   More actions

   Update: thread per task in production

   One replica ran with a thread per task (switched with PATCH /settings), and the other stayed on the shared pool as the control, for about an hour of a slow period. The switched replica carried about 1.4 times the tasks of the control.

   	shared pool	thread per task
   queue mean / p99 / p99.9, first step	7.2 / 204 / 657 ms	1.8 / 45 / 202 ms
   tasks that waited in the queue > 0.4 s	2,070	29
   tasks that ran > 3.3 s / > 6.5 s	66 / 13	10 / 0
   run p99.9, first step	556 ms	400 ms
   streams > 80 ms	3.8%	1.7%

   • Queue waits over 0.4 s almost stopped.
   • The run-time tail fell too. On the shared pool, the run time of a task includes the tasks that its worker picked up while it waited in join. Per task, runs over 1.6 s were as frequent as on the control, and runs over 3.3 s were about 9 times less frequent.
   • The most frequent heavy query shape needs about 3.2 times more own time per block on one thread, close to the 3.6× of the model. Even so, no task on that replica ran over 6.5 s.
   • CPU per task was lower on the switched replica.
   • No client retries, 5xx, too many tasks or worker panics on either replica.

   Both replicas now run with a thread per task.

   What the slow-task log (#117) shows

   • Most slow tasks are the first step of a stream that starts hundreds to thousands of blocks behind the head, on the dataset kind with the widest tables. The filters are selective, and most of these queries ask for several relations.
   • About 60% of these tasks buffer less than 50 KB. The cost is in the scans over whole chunks, not in the output.
   • A stream that catches up reads every chunk in full anyway, so applying the block range in the scans (hypothesis 2) would save only a part of the first chunk. Hypothesis 1 holds in this form: selective filters with relations over large tables.

   Next

   1. Merge feat(hotblocks): run each query task on its own thread behind a flag #119.
   2. Profile the scans of these query shapes. On one thread, each of them now costs only its own latency, but it still takes the most CPU per block.