Repository navigation
Plan a compiled write's per-row reads as nested loops on Postgres, so a MERGE over a table vacuumed while empty isn't quadratic (#420) - #428
Merged
Conversation
… a MERGE over a table vacuumed while empty isn't quadratic (#420) Port of liberation-data/drivine4j#609. A write's top-level loop that reads the node or edge table for each row, where it adds to that table or joins it to a CALL's rows, saves enable_seqscan, enable_hashjoin and enable_mergejoin, turns them off for the loop (set_config(..., true)) and puts back what they were after it. The timing tests' MERGE and CALL...RETURN cases on Postgres pass again, without the xfail marks. Oracle and SQLite compile as before.
… failure before it can't pass for the restore (#420) The case caught any error, so one before the loop turned the settings off would have read them back unchanged and passed. It now asserts the loop's "division by zero". Same gap as liberation-data/drivine4j#614.
This was referenced Nov 5, 2026
…ACUUM's shared memory doesn't starve a later vector-index build (#420) The test vacuumed the shared edge table with about 20k dead relationships, which Postgres does in parallel. The pooled session that ran it kept about 1.2 MB of /dev/shm until its pool closed, and a later parallel HNSW build in the sync schema and search tests could not get its 64 MB block of a 64 MB /dev/shm. The writes now run on cy_keyed, as in drivine4j and TypeScript, and each case deletes its rows and ends its pool.
This was referenced Nov 5, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #420. Port of liberation-data/drivine4j#609, merged there as liberation-data/drivine4j#613. The TypeScript port is liberation-data/drivine#628.
Cause
On Postgres, a compiled write is a PL/pgSQL function. Its loop over the statement's rows reads the node or edge table once for each row. Two things planned that read as a Seq Scan, which made the loop quadratic:
CALL { … }'s rows,jsonb_array_elements(cy_rows), at 100, so it joined them to the node table with a hash join over a Seq Scan.The fix
PostgresCypherCompiler.procedurewraps each top-level loop whose body both:jsonb_array_elements(.Before the loop, it saves
enable_seqscan,enable_hashjoinandenable_mergejoinintocy_planner text[]and turns them off withset_config(…, 'off', true). AfterEND LOOPit sets them back to the saved values. Because the settings are set for the transaction, a statement that fails in its loop leaves them as its transaction or savepoint had them.The compiled SQL
The Cypher spec has 1,297 statements on Postgres:
tests/golden/statements.jsonholds no compiled procedures.The xfails are gone
tests/integration/sql/test_compiled_write_loop_time.pyand its sync twin no longer mark the Postgres MERGE case (from #421) or the CALL…RETURN case (from #425) as xfail. Both pass as plain tests, at 2k → 20k rows:Tests
tests/unit/test_postgres_nested_loop_plan.py:tests/integration/postgres/test_postgres_nested_loop_settings.py, and its sync twin. The settings are restored:SET;SET LOCAL enable_hashjoin = off;The loop is made to fail with
n.i / 0, not drivine4j's null-property relationship MERGE, because that MERGE doesn't raise here (Cypher on SQL stores: MERGE with a null property makes a new node each run, where Neo4j refuses it #380).tests/integration/postgres/test_compiled_write_keyed_read_time.py, and its sync twin: drivine4j's 7 shapes at 2k and 20k rows, all 0.55–0.86 s at 20k.tests/unit/test_compiled_write_kept_rows.py: the Postgres case for a procedure that keeps no rows now ignores the newcy_planner text[];line, since itsCALL (n) { DETACH DELETE n }loop qualifies.Docs
docs/04-statements.mdhas a Postgres bullet, the same as drivine4j's.Gate
The following pass:
ruff check,ruff format --check,pyrightandunasync --check;test_cypher_write_loop.py, async and sync.Two review passes ran, and both were clean.
The CI failure on this PR's first runs
The first CI runs failed in the sync vector-index tests with
DiskFull: could not resize shared memory segment … to 63999616 bytes. The new keyed-read test was the cause:/dev/shmuntil its pool closed./dev/shm.The test now runs on a graph of its own,
cy_keyed, as drivine4j and TypeScript do. Each case deletes its rows and ends its pool. The full suite passes this way, with no change to the container's shm size.Docs follow-up, not in this PR. A user's own large table can hit the same limit. Once a table passes 8 MB, Postgres builds a pgvector HNSW index on it in parallel. That build asks for almost all of
maintenance_work_memas one block of/dev/shm. With 64 MB of/dev/shm, Docker's default, it fails withDiskFullas soon as anything else holds a little of it.