Skip to content

add sql formatter and fix parser gaps - #34

Open
emrberk wants to merge 7 commits into
masterfrom
feat/formatter
Open

emrberk wants to merge 7 commits into
masterfrom
feat/formatter

Conversation

@emrberk

@emrberk emrberk commented Sep 18, 2026

Copy link
Copy Markdown
Member

Formatter

Adds format(sql, options) — a token-based SQL formatter, exported from the root and from @questdb/sql-parser/formatter (lexer only, no parser in the bundle).

  • Reuses the parser's token definitions through a second, lossless lexer. Whitespace and comments are kept as trivia, unknown characters become opaque tokens, unterminated strings and comments are tolerated.
  • Never throws on SQL content. Every token, comment and unknown character is preserved; only whitespace between tokens changes. Unbalanced or unterminated input is preserved verbatim from the point of the problem.
  • Layout is driven by a small phrase table per statement kind (SELECT, INSERT, UPDATE, CREATE TABLE/MATERIALIZED VIEW/LIVE VIEW, ALTER TABLE/MATERIALIZED VIEW). Unknown statements get normalized spacing and no line breaks. One width rule (maxLineWidth, default 80): lists, predicates, CASE, IN/VALUES lists, OVER (...), named windows, join sub-clauses and PIVOT (...) stay inline when they fit, otherwise break one item per line.
  • Options: indent (default two spaces), maxLineWidth.
  • Tests: 63 exact-output fixtures; the full docs corpus and a seeded malformed-input corpus (truncations, delimiter edits, injected garbage) checked against four oracles — identical token stream, identical operator adjacency, idempotence, and identical AST where the parser accepts the input; performance budget for a 5,000-line script and 50-level nesting.

Playground: https://questdb-sql-formatter-playground.netlify.app/

Parser

Gaps found by running the current documentation through the parser and the round-trip test:

  • ALTER USER | GROUP | SERVICE ACCOUNT … SET MEMORY LIMIT <size> | UNLIMITED
  • RIGHT [OUTER] JOIN, FULL [OUTER] JOIN
  • UNNEST(...) after a join, e.g. CROSS JOIN UNNEST(t.asks[2]) u(vol)
  • FILL(PREV(other_column), PREV)
  • CONVERT PARTITION TO PARQUET … WITH (bloom_filter_columns = …, bloom_filter_fpp = …)
  • CREATE LIVE VIEW … OWNED BY owner
  • FROM t alias TIMESTAMP(col) (alias before the designation, as the server parses it)
  • COPY … TO … WITH … BLOOM_FILTER_COLUMNS … BLOOM_FILTER_FPP …
  • ALTER MATERIALIZED VIEW … ADD INDEX TYPE POSTING [DELTA | EF] [INCLUDE (…)] [CAPACITY n]

Serializer: CREATE LIVE VIEW … AS SELECT no longer gains parentheses on output; CREATE TABLE writes FORMAT after WAL.

Corpus

docs-queries.json grows from 1,797 to 2,637 queries: every questdb-sql block in the documentation that was not yet covered (syntax templates, fragments and intentionally invalid examples excluded), plus the queries from the 0.1.16 verification set. All pass parse, round-trip, autocomplete walkthrough and the formatter oracles.

No version bump.

emrberk and others added 7 commits September 18, 2026 11:12
PR #33 added memory_limit, cpu_weight, max_active_queries,
max_queued_queries, queue_timeout, mapping, priority, resource and
unset as tokens without adding them to IDENTIFIER_KEYWORD_NAMES, so
they could no longer be used as column names. memory_limit is a column
of query_activity(), which the Web Console queries.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…th 50

- Indent the query one level after AS in CREATE statements and after
  the INSERT target.
- Open a block for a parenthesis that follows FROM, JOIN, a comma in the
  FROM list, or AS in a WITH item or CREATE, so implicit-select
  subqueries break into lines; a single plain item stays inline.
- Drop gaps at the start of a line in the printer, which keeps blocks
  that begin with a comment or unknown text idempotent.
- DECLARE with two or more variables always breaks, one per line.
- ATTACH/DETACH/DROP/CONVERT PARTITION LIST keep LIST in the clause head.
- Default maxLineWidth is 50; fixtures pin width 80 unless they set it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two changes that both come from the same idea: the grammar is the source
of truth, so the formatter should read it rather than repeat it.

Grammar drift test. The phrase table that drives layout is hand-written
while QuestDB syntax arrives through the parser, so the two drift apart
silently and an unknown clause stays inline instead of starting a line.
tests/formatter/grammar-drift.test.ts reads the grammar through
Chevrotain and fails when a statement rule can start a part with a
keyword sequence the formatter neither breaks on nor lists as inline by
design, and when a phrase in the table no longer exists in the grammar.
It found six gaps, fixed here with fixtures:

- LATEST BY, the legacy form of LATEST ON, was not a clause
- UPDATE ... FROM t JOIN u did not break its joins
- ALTER TABLE SUSPEND WAL, REBASE WAL and the storage policy actions
  stayed inline while ADD COLUMN started a line
- ALTER MATERIALIZED VIEW DROP EXPIRE stayed inline
- EXPIRE ROWS was not a CREATE option
- UNPIVOT was in the table but never in the grammar

Capitalize option. format(sql, { capitalize: true }) uppercases the
keywords. Casing cannot be decided from the word alone, since QuestDB
has about 60 reserved keywords and 350 non-reserved ones, and the
non-reserved group holds most type and option words, which are also the
most common column names. So it parses the statement and reads the
answer off the tree: a word the grammar consumed through its identifier
rule names a table, view or column and keeps its case, every other
keyword is raised, and SQL that does not parse keeps the case it was
written in. Layout stays the same either way and needs no lists.

The corpus test formats every parseable query this way and requires that
the words the grammar read as names come back with their exact case.

maxLineWidth is renamed to maxWidth.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 5,000-line budget failed on CI at 98 ms against 80 ms, while the same
script measures 22 ms locally and 23 ms on the commit before the
formatter changes, so nothing had slowed down. The budget was four times
a local median taken before the corpus grew, and this file runs beside
suites that saturate a shared runner.

Assert how the cost grows instead, which no machine speed can move: ten
times the script must cost less than twenty five times the work, and
four times the nesting less than twelve times. Measured today those are
7.5 to 9.4 times and 2.4 times. A loose ceiling still catches a hang.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant