Skip to content

Plan a compiled write's per-row loop on Postgres without materializing, so a node table whose statistics say it is near empty isn't read whole for each row (#430) - #441

Merged
jasperblues merged 1 commit into
mainfrom
fix-430-call-return
Nov 6, 2026
Merged

jasperblues merged 1 commit into
mainfrom
fix-430-call-return

Conversation

@jasperblues

Copy link
Copy Markdown
Member

Closes #430. The reference fix is liberation-data/drivine4j#631; the TypeScript port is liberation-data/drivine#646.

Cause

#420 turned off enable_seqscan, enable_hashjoin and enable_mergejoin around a compiled write's per-row loop on Postgres. An autoanalyze that lands just after a DELETE leaves the node table's statistics at about one row, while the table still holds many pages. In that state, the loop's per-row join of a CALL's rows is planned like this:

Nested Loop Left Join  (rows=100)
  Join Filter: (n8.nid = ((r7.item ->> 'v1'::text))::bigint)
  Rows Removed by Join Filter: 17356
  ->  Function Scan on jsonb_array_elements r7  (rows=100)
  ->  Materialize  (actual rows=17357)
        ->  Index Scan using drivine_node_pkey on drivine_node n8  (rows=1) (actual rows=17357)

That is a full, unparameterized scan of the primary-key index under a Materialize, so every row reads the whole table. MATCH (n:CyLoop) CALL (n) { SET n.y = n.i RETURN n.i AS j } RETURN count(j) took between 72 s and 142 s over 20,000 rows. It failed in 3 of about 4 CI runs, sync and async alike. On a test machine it happened 3 times in 400 runs, with auto_explain capturing the plan above.

The fix

enable_material joins _JOIN_METHODS. Each wrapped loop now also runs with materialization off, so it reads the index by the row's key, and the setting is put back with the other three.

Tests

  • test_compiled_write_loop_time.py, async and sync: on Postgres, the test runs ANALYZE drivine_node right after its DELETE and turns autovacuum off on that table while the writes are timed (reset in finally). Without the guard, an autoanalyze landing mid-run sometimes reset the statistics. With the bad state forced, the test fails every time without the fix (about 98 s over 20,000 rows) and passes with it (about 0.65 s).
  • test_postgres_nested_loop_plan.py and test_postgres_nested_loop_settings.py: they cover the fourth setting, including its restore when the loop fails.

Docs

docs/04-statements.md names enable_material.

Gate

  • ruff check, ruff format --check, pyright and unasync --check pass.
  • The unit suite passes.
  • The Postgres integration suites pass, together with the sync schema suites, through the test lock.
  • Two review passes ran; the second was clean.

…g, so a node table whose statistics say it is near empty isn't read whole for each row (#430)

#420 turned off seq scans and hash and merge joins around a compiled
write's per-row loop. Once an autoanalyze just after a DELETE left the node
table's statistics at about one row while it still held many pages, a
CALL's rows were joined to it through a Materialize over a full scan of
its primary-key index, so each row read the whole table: CALL...RETURN
took 72 s to 142 s over 20,000 rows on CI. The loop now runs with
enable_material off too, and the loop-time test forces that state.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A compiled CALL…RETURN over 20,000 rows took 142 s once on CI (Python 3.14, sync), after #420's fix

1 participant