Skip to content

fix(treesitter, cli): no recursion in QueryResultIter, explicit worker stack size - #944

Open
psteinroe wants to merge 2 commits into
mainfrom
fix/treesitter-query-iter-recursion
Open

psteinroe wants to merge 2 commits into
mainfrom
fix/treesitter-query-iter-recursion

Conversation

@psteinroe

Copy link
Copy Markdown
Collaborator

Two stack fixes, one commit each: a recursion that grows with the input, and the thread stack size that turns such recursions into an abort of the daemon.

No recursion in QueryResultIter

QueryResultIter::next (pgls_treesitter/src/queries/mod.rs) called itself once for every query result outside the requested range, the same pattern as the ParseIterator fixed in #892. The completion and hover context filters relations, aliases and columns by the range of the statement, so a long statement could overflow the stack. It is a find now.

Test: skips_many_results_outside_the_range filters 50,000 relations down to the last one on a thread with a 2 MiB stack. It aborts with a stack overflow on main.

An explicit stack size for the worker threads

The daemon's tokio runtime (which runs the language server's handlers and its spawn_blocking threads) and the rayon pool of check used the 2 MiB default. A stack overflow aborts the process, so for the daemon it kills the server for every connected editor. That is what turned R1 from #892 into a crash.

Not every deep recursion can be removed: libpg_query serializes the parse tree recursively, with stack proportional to the depth of the tree. A left-deep expression such as select 1 + abs(1) + abs(1) + ... is deep while staying valid SQL. Measured on this machine with hover, inlay hints, diagnostics, code actions and formatting on that statement, in a release build:

terms 2 MiB 4 MiB 8 MiB 16 MiB
5,000 ok ok ok ok
10,000 overflow ok ok ok
20,000 overflow overflow ok ok
50,000 overflow overflow overflow overflow

Postgres 15 with its default max_stack_depth of 2 MB accepts 4,000 terms and rejects 5,000 ("stack depth limit exceeded"). With WORKER_STACK_SIZE_BYTES = 8 MiB, the default stack of the main thread on Linux, the server handles four times what Postgres accepts by default. The new pgls_cli::runtime() builds every tokio runtime of the CLI and the daemon with it, and the rayon pool of check uses it too. The memory is virtual and only committed as it is used, so idle threads don't cost 8 MiB each.

Test: workers_parse_long_expressions parses a 2,000-term expression on a worker of runtime(). In a debug build, where libpg_query's frames are larger, that already overflows tokio's default 2 MiB.

Not covered

  • A 50,000-term expression still overflows 8 MiB. Bounding that would need a depth check in libpg_query's serializer, or checking the depth before parsing. These expressions are far beyond what Postgres accepts.
  • The decode of such trees fails anyway, because prost's recursion limit rejects them (P22, another lane). They are reported as syntax errors.
  • The runtime in pgls_workspace's async_helper only runs database futures and keeps the default.

QueryResultIter::next recursed once for every query result outside the
range, so a long statement overflowed the stack, like ParseIterator did.
The language server's tokio threads and the rayon threads of check used
the 2 MiB default. libpg_query needs stack proportional to the length of
an expression, and an overflow aborts the daemon for every editor. The
threads now get 8 MiB, the main thread's default on Linux.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant