[psteinroe]

Stacked on #956. The first commit is the fix from #963 (a panic on a trailing WITH), which the randomized test below found. It drops out when #963 is merged.

Every change_file split the whole document again (lexer, splitter, and the blank-line recovery pass) and scanned every line for suppression comments. That is linear in the file and runs on every keystroke:

change_file, one keystroke in the middle of the file	before	after
1,000 statements (89 KB), release	1.42 ms	0.032 ms
20,000 statements (1.8 MB), release	29.5 ms	0.56 ms
1,000 statements, debug	33.6 ms	0.56 ms
20,000 statements, debug	661 ms	9.6 ms

Measured with a throwaway example (not committed) that types and deletes a character in a table name in the middle of a migration-style file, 200 times, through WorkspaceServer::change_file.

Incremental split

pgls_statement_splitter::split_edited(old, previous, new) returns the same result as split(new):

1. Find the change. The common prefix and suffix of the old and new text give the changed part, compared in 64-byte blocks. Several edits at once become one changed part.

2. Re-split a region. The region runs from EDIT_CONTEXT_STATEMENTS = 2 statements before the change to as many after it, split on its own with split. The statements before the region are kept, and the ones after it are shifted by the change in length. Errors are kept and shifted the same way, lexer errors first and then splitter errors, as split orders them.

3. Check the edges. The two statements at each edge of the region have to come out exactly as in the previous split. That shows the region lines up with the statements around it:

   • the splitter decides statement boundaries from the tokens of the statement and its neighbours (look_back, look_ahead);
   • the blank-line recovery pass works per coarse statement;
   • the lexer is stateless outside of strings, comments and dollar quotes, which the next point excludes.

   The first and the last statement of the region, the ones that are split without their real neighbours (no look_back at the start, the end of the region instead of the next statement), are taken from the previous split as well. Without that, an error that a statement reports at the end of the text (Expected FROM_KW on delete followed by EOF) leaked into the result.

4. Fall back to a full split when the change, or one byte on either side of it, touches $ ' " / * - \. Those can open or close a string, a dollar quote, a comment or a psql command, and so change how everything after them is lexed. A full split is also used when the edge statements don't match.

The Document keeps the SplitResult of its content and passes it to split_edited in update_content. SplitResult derives Clone/PartialEq and records how many of its errors come from the lexer.

Suppressions

Suppressions::from now returns early when the text doesn't contain -ignore. Every suppression comment contains it, and the check is a single substring search. Without line suppressions the line index isn't needed, so it isn't built either. This took 2.5 ms of the remaining 3 ms at 20,000 statements. With suppressions in the file, the parser still runs over all lines as before.

Two small fixes in the same code:

• is_suppressed computed the lines of a diagnostic even without any line suppression;
• LineIndex::line_for_offset was a linear scan over all lines. It is a binary search now, which matters when a file with suppressions has many diagnostics.

Tests

• split_edited_is_split: 3,000 random edits over 10 documents built from statements the splitter treats differently: $$ and BEGIN ATOMIC functions, comments, quoted ;, blank lines inside statements, statements without ;, invalid SQL. The edits insert, delete or replace text, including fragments that cut or merge statements (;, \n\n, select, begin atomic, ', $$, /*, --, \d), sometimes two at once. After each edit the result must equal split(new), ranges and errors included. Each edit starts from the previous incremental result, so errors would accumulate. SPLIT_SEED and SPLIT_EDITS change the run; locally, 12 seeds × 50,000 edits passed in release. About 43% of these edits are split incrementally; the rest touch a quote or comment and fall back. Normal typing (letters, spaces, ;, parentheses) is incremental.
• updates_split_like_the_whole_document (workspace): 1,000 edits through Document::update_content. After each, the document's split and syntax diagnostics equal those of a full split.
• documents_without_suppressions_skip_the_lines: the fast path gives the same suppressions as parsing the lines.
• The edit-churn and memory tests from perf(workspace): bound the memory of the document caches #951/test(workspace): the document caches follow a long editing session #956 pass unchanged. The churn test drops from 40 s to 33 s in debug.

Not covered

• Changes inside strings, comments and dollar-quoted bodies fall back to a full split. Typing inside a SQL function body therefore still costs a full split. Handling it would need the lexer to resume from a known state.
• The pruning of the statement caches in update_content still hashes the text of every statement, about 0.3 ms at 20,000 statements in release. It could use the changed region too.