Avoid quadratic accumulation when parsing long SQL expression lists

[arc-oai]

Summary

• Replace repeated array concatenation in SQL expression lists with persistent linked lists and defer array materialization until completed expressions are needed.
• Preserve grammar ambiguity handling and formatted output while covering large IN lists, long OR chains, and wide SELECT lists.
• Re-enable the existing memory regression for Memory issue with large queries #840.

Why

Nearley expands free_form_sql:* into an append postprocessor equivalent to items.concat([item]). Repeatedly copying every prefix produces quadratic allocation. For a representative 1,000-value IN query, these generated repetition postprocessors copied 1,997,014 prefix entries before this change and 10 afterward. Final AST construction still performs the necessary linear work: 3,004 list entries are materialized and another 3,000 entries are copied into completed arrays. Mutating the shared array with .push() is unsafe because alternative parses share prefixes.

free_form_sql remains the existing rule for one SQL element. The new expression_list recognizes exactly the same zero-or-more sequence as free_form_sql:*:

expression_list -> null
expression_list -> expression_list free_form_sql

Each append instead creates an immutable { previous, value } node. Consequently, expressions_or_clauses -> expression_list clause:* preserves the original grammar: free-form elements followed by structured clauses. Clause materialization is deferred because eagerly converting every intermediate parser candidate would reintroduce quadratic copying.

Benchmarks

Apple M4 Max, Node.js 24.12.0. Values show median formatting time and whole-process peak RSS. Stress processes used a 384 MB V8 old-space limit.

Input	Before	After
Simple 9 KB query	4.19 ms / 108 MB	4.08 ms / 111 MB
1,000-value IN	21.35 ms / 190 MB	13.07 ms / 90 MB
3,000-value IN	107.25 ms / 444 MB	34.47 ms / 109 MB
10,000-value IN	Out of memory	97.98 ms / 157 MB
5,000-condition OR chain	Out of memory	104.61 ms / 157 MB
10,000-column SELECT	Out of memory	110.04 ms / 229 MB

Validation

• pnpm run grammar
• pnpm run ts:check
• pnpm run pretty:check
• pnpm run lint
• pnpm run build
• pnpm exec jest --runInBand --silent --coverage=false: 27 suites, 5,841 tests, and 63 snapshots passed.
• Re-enabled issue Memory issue with large queries #840 performance regression passed.
• 1,000 generated differential SQL cases preserved existing output and rejection behavior.

Closes #840.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cf9be9cf-006c-4a3a-b629-07bc6d385f39

📥 Commits

Reviewing files that changed from the base of the PR and between aa8efae and 6544625.

📒 Files selected for processing (2)

• src/parser/grammar.ne
• test/perftest.ts

---

📝 Walkthrough

Summary by CodeRabbit

• Performance

  • Improved SQL parsing efficiency by reducing intermediate memory usage while preserving expression and clause order.
  • Large SQL formatting workloads now receive more efficient processing, including files around 100 KB.

• Reliability

  • Improved handling of expressions, clauses, set operations, and nested structures during parsing.

Walkthrough

The parser now accumulates expressions with linked lists and materializes them when building AST children. Clause and set-operation children use lazy getters. The large-query memory test is enabled.

Changes

Parser expression handling

Layer / File(s)	Summary
Expression list helpers src/parser/grammar.ne	Linked-list helpers accumulate expressions and materialize ordered arrays with optional clauses.
Grammar integration and memory validation src/parser/grammar.ne, test/perftest.ts	Statements, clauses, set operations, and grouped expressions use expression lists and lazy child getters. The approximately 100 KB SQL memory test now runs.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: preventing quadratic accumulation when parsing long SQL expression lists.
Description check	✅ Passed	The description explains the linked-list optimization, affected SQL cases, validation, benchmarks, and issue #840 regression.
Linked Issues check	✅ Passed	The changes directly address issue #840 by reducing memory usage for large IN clauses, OR chains, and SELECT lists.
Out of Scope Changes check	✅ Passed	The grammar optimization and re-enabled performance test are directly related to the linked issue and stated pull request objectives.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[nene]

Thanks for the PR. The fix seems to work, but the code is somewhat cryptic and seems to contain various extra stuff that's not really part of the core optimization proposed in this PR.

[nene]

What's the purpose of this wrapping of children property into a getter function?

It looks like some an additional performance optimization. But not sure it's really adding anything. At least when I removed these, the code still seemed to perform similarly.

If this is a separate unrelated optimization, then please make a separate PR with it. If it's tightly related to the current optimization and really needed for it, please provide some explanation.

[arc-oai]

Yeah this does help in cases like SELECT column_1, column_2, column_3, ..., column_1000 FROM my_table or SELECT * FROM my_table WHERE id = 1 OR id = 2 OR id = 3 ... OR id = 1000;. Nearley creates an intermediate clause for every prefix, so eagerly constructing children repeatedly copies the entire prefix and makes this quadratic.

The test from eb187fe mostly demonstrates this case. It does "pass" without this change because peak/final memory is lowered (by only the linkedlist change), but it still takes ~20x longer without this (using quadratic total memory, even if not peak memory)

I could split this into a separate follow-up PR if that helps, was just broadly tackling the several related issues of "quadratic costs"

[nene]

OK. Thanks for the explanation. Makes sense now.

[nene]

Thanks for the cleanup and clarifications.