Skip to content

Plan a compiled write's per-row reads as nested loops on Postgres, so a MERGE over a table vacuumed while empty isn't quadratic (#420) - #428

Merged
jasperblues merged 3 commits into
mainfrom
fix-420-merge-plan
Nov 5, 2026
Merged

jasperblues merged 3 commits into
mainfrom
fix-420-merge-plan

Conversation

@jasperblues

@jasperblues jasperblues commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Closes #420. Port of liberation-data/drivine4j#609, merged there as liberation-data/drivine4j#613. The TypeScript port is liberation-data/drivine#628.

Cause

On Postgres, a compiled write is a PL/pgSQL function. Its loop over the statement's rows reads the node or edge table once for each row. Two things planned that read as a Seq Scan, which made the loop quadratic:

  1. A plan made while the table was empty. Once the table had been vacuumed or analysed while empty, PL/pgSQL kept a generic plan that scans it.
  2. A CALL's row estimate. The planner estimated a CALL { … }'s rows, jsonb_array_elements(cy_rows), at 100, so it joined them to the node table with a hash join over a Seq Scan.

The fix

PostgresCypherCompiler.procedure wraps each top-level loop whose body both:

  • reads the node or edge table, and
  • inserts into that table, or joins jsonb_array_elements(.

Before the loop, it saves enable_seqscan, enable_hashjoin and enable_mergejoin into cy_planner text[] and turns them off with set_config(…, 'off', true). After END LOOP it sets them back to the saved values. Because the settings are set for the transaction, a statement that fails in its loop leaves them as its transaction or savepoint had them.

  • Same text as drivine4j: the rule, the regexes and the emitted text match drivine4j's line for line.
  • Unchanged: Oracle and SQLite compile as before.

The compiled SQL

The Cypher spec has 1,297 statements on Postgres:

  • Postgres: 53 change, the same count as drivine4j. Masking the planner lines and the function-name hash, each is identical to before.
  • Oracle: byte-identical.
  • SQLite: byte-identical apart from the compile-time clock.
  • Golden statements: no golden statement is affected, because tests/golden/statements.json holds no compiled procedures.

The xfails are gone

tests/integration/sql/test_compiled_write_loop_time.py and its sync twin no longer mark the Postgres MERGE case (from #421) or the CALL…RETURN case (from #425) as xfail. Both pass as plain tests, at 2k → 20k rows:

Case async sync
MERGE, with the edge table vacuumed empty 52 → 495 ms 62 → 507 ms
CALL…RETURN 65 → 606 ms 65 → 672 ms

Tests

  • tests/unit/test_postgres_nested_loop_plan.py:

    • which loops get the settings (7 that do, 6 that don't);
    • a statement with two qualifying loops;
    • Oracle and SQLite unchanged.
  • tests/integration/postgres/test_postgres_nested_loop_settings.py, and its sync twin. The settings are restored:

    • after a session-level SET;
    • after a transaction-level SET LOCAL enable_hashjoin = off;
    • when the loop raises and the savepoint is rolled back.

    The loop is made to fail with n.i / 0, not drivine4j's null-property relationship MERGE, because that MERGE doesn't raise here (Cypher on SQL stores: MERGE with a null property makes a new node each run, where Neo4j refuses it #380).

  • tests/integration/postgres/test_compiled_write_keyed_read_time.py, and its sync twin: drivine4j's 7 shapes at 2k and 20k rows, all 0.55–0.86 s at 20k.

  • tests/unit/test_compiled_write_kept_rows.py: the Postgres case for a procedure that keeps no rows now ignores the new cy_planner text[]; line, since its CALL (n) { DETACH DELETE n } loop qualifies.

Docs

docs/04-statements.md has a Postgres bullet, the same as drivine4j's.

Gate

The following pass:

  • ruff check, ruff format --check, pyright and unasync --check;
  • the unit suite;
  • the Postgres integration tests above, together with test_cypher_write_loop.py, async and sync.

Two review passes ran, and both were clean.

The CI failure on this PR's first runs

The first CI runs failed in the sync vector-index tests with DiskFull: could not resize shared memory segment … to 63999616 bytes. The new keyed-read test was the cause:

  1. It vacuumed the shared edge table while that table held about 20k dead relationships. Postgres runs that vacuum in parallel.
  2. The pooled session that ran it kept about 1.2 MB of /dev/shm until its pool closed.
  3. A later parallel HNSW build then couldn't get its 64 MB block out of Docker's 64 MB /dev/shm.

The test now runs on a graph of its own, cy_keyed, as drivine4j and TypeScript do. Each case deletes its rows and ends its pool. The full suite passes this way, with no change to the container's shm size.

Docs follow-up, not in this PR. A user's own large table can hit the same limit. Once a table passes 8 MB, Postgres builds a pgvector HNSW index on it in parallel. That build asks for almost all of maintenance_work_mem as one block of /dev/shm. With 64 MB of /dev/shm, Docker's default, it fails with DiskFull as soon as anything else holds a little of it.

… a MERGE over a table vacuumed while empty isn't quadratic (#420)

Port of liberation-data/drivine4j#609. A write's top-level loop that reads
the node or edge table for each row, where it adds to that table or joins
it to a CALL's rows, saves enable_seqscan, enable_hashjoin and
enable_mergejoin, turns them off for the loop (set_config(..., true)) and
puts back what they were after it. The timing tests' MERGE and
CALL...RETURN cases on Postgres pass again, without the xfail marks.

Oracle and SQLite compile as before.
… failure before it can't pass for the restore (#420)

The case caught any error, so one before the loop turned the settings off
would have read them back unchanged and passed. It now asserts the loop's
"division by zero". Same gap as liberation-data/drivine4j#614.
…ACUUM's shared memory doesn't starve a later vector-index build (#420)

The test vacuumed the shared edge table with about 20k dead relationships,
which Postgres does in parallel. The pooled session that ran it kept about
1.2 MB of /dev/shm until its pool closed, and a later parallel HNSW build
in the sync schema and search tests could not get its 64 MB block of a
64 MB /dev/shm. The writes now run on cy_keyed, as in drivine4j and
TypeScript, and each case deletes its rows and ends its pool.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A compiled MERGE of many relationships is quadratic on Postgres once the edge table is vacuumed empty

1 participant