Skip to content

Score a prefix term the same for each node that holds it, as Lucene's prefix query does, on the SQL stores (#427) - #433

Draft
jasperblues wants to merge 2 commits into
mainfrom
fix-427-prefix-constant
Draft

jasperblues wants to merge 2 commits into
mainfrom
fix-427-prefix-constant

Conversation

@jasperblues

Copy link
Copy Markdown
Member

Closes #427. Port of liberation-data/drivine4j#612 (drivine4j PR #623), as liberation-data/drivine#638 ports it. Not to merge until train 2026.11.1 is cut, and not before drivine#638.

On Postgres, SQLite and Oracle without Oracle Text, a prefix term (kiw*) was weighed by BM25, while Neo4j's Lucene scores a prefix query as a constant. bm25_ranked (drivine/backend/sql/bm25.py) now adds PREFIX, k1 + 1 = 2.2, to each node that holds a prefix term (is_prefix). That is Lucene's constant of 1 on bm25_score's scale. Every other term scores as before.

Goldens: unchanged, since none of them pins a prefix query.

Tests

  • tests/integration/test_bm25.py, and its sync copy, on Neo4j, Postgres, Postgres indexed, SQLite and Oracle:
    • kiw* scores every hit 1.0;
    • kiw* bird ranks the node with both first;
    • a Fruit label with #612's texts: kiw* and kiwi* give every hit 1.0, and kiw* melon and +kiw* melon give the formula's figure, within 0.01 of Neo4j's measured 0.5409. On Neo4j, with lengths in tokens, it's within 1e-4.
  • tests/unit/test_bm25.py: the score SQL for kiw* melon.

Verification

  • Two fresh-reviewer passes. Pass 1 added the unit case and the docs notes (docs/02-connecting.md, docs/07-schema.md); pass 2 came back clean.
  • Unit tests: 2,471 passed. ruff and pyright are clean, and the sync code is current.
  • Integration, on the test box with Oracle: the BM25 tests, their sync copy, and the search tests. 207 passed, and agreement with Neo4j is unchanged at 17/21.

… prefix query does, on the SQL stores (#427)

Port of liberation-data/drivine4j#612 (drivine4j PR #623). A prefix term adds k1 + 1 to each node
that holds it: Lucene's constant of 1 on the scale of bm25_score.
…that a prefix term adds the same to each node (#427)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Full-text search weighs a prefix term (kiw*) by BM25, where Neo4j scores every prefix hit the same

1 participant