Skip to content

Count a term written again at once each time, not every other time, on the SQL stores (#422) - #431

Draft
jasperblues wants to merge 3 commits into
mainfrom
fix-422-adjacent
Draft

jasperblues wants to merge 3 commits into
mainfrom
fix-422-adjacent

Conversation

@jasperblues

Copy link
Copy Markdown
Member

Closes #422. Port of liberation-data/drivine4j#607 (drivine4j PR #619), with the statements of liberation-data/drivine#637 byte for byte. Not to merge until train 2026.11.1 is cut, and not before drivine#637. Once #637 merges, re-record tests/golden/VERSION with scripts/sync_golden.py. The statements already match its branch byte for byte.

On Postgres, SQLite and Oracle without Oracle Text, a full-text term written again at once was counted every other time: "kiwi kiwi kiwi" counted 2. Since BM25 (#423), that undercount reaches the ranking.

How each store counts now

This follows drivine4j#619 and drivine#637 (drivine/backend/sql/text_query.py: looking_ahead, counted, SEPARATOR).

  • Postgres and SQLite: the term's trailing boundary is a lookahead, (?=$|[^[:alnum:]_]).

  • Oracle, in oracle_search._oracle_count, by one of three ways:

    • Most terms: two counts, the term ending where its last word does minus the same followed by [[:alnum:]_].
    • A phrase that repeats a word or whose last word starts its first: its own expression over the text with every separator doubled.
    • Prefix terms, and terms holding a number that isn't a digit: the one expression, as before.

    A word is letters (isalpha), decimal digits (isdecimal) and _, which is \p{L}, \p{Nd} and _, as in Java and TypeScript.

Tests

  • tests/integration/test_bm25.py, and its sync copy, on Neo4j, Postgres, Postgres indexed, SQLite and Oracle:
    • "okapi okapi okapi" outranks "okapi lemur okapi";
    • +"la la" finds "la la land";
    • +"tapirs tapir" finds "tapirs tapirs tapir";
    • many's pinned score is now 0.1359.
  • tests/unit/test_text_query_counting.py: each of drivine4j's cases counted both ways through Patterns, with exact counts.
  • tests/unit/test_bm25.py: the SQL and bindings for each of Oracle's three ways.
  • Goldens: the Oracle *.matching statements, copied from drivine#637. Byte-identical.

Verification

  • Three fresh-reviewer passes. Pass 1 added the exact-count and Oracle SQL tests and the docs notes, pass 2 a docstring, and pass 3 came back clean.
  • Unit tests: 2,480 passed. ruff and pyright are clean, and the sync code is current.
  • Integration, on the test box with Oracle: the BM25 tests and their sync copy, the search tests, and the Postgres, SQLite and Oracle store and schema tests. 239 passed, and agreement with Neo4j is 17/21 on every SQL store.

…n Postgres, SQLite and Oracle (#422)

Port of liberation-data/drivine4j#607 (drivine4j PR #619), with the statements of
liberation-data/drivine#629 byte for byte. Postgres and SQLite hold to the character after a term
without taking it. Oracle counts the term ending where its last word does, less those a word goes on
from; a phrase whose occurrences may begin inside one another in the text with its gaps doubled; and
a prefix term, or one holding a number that isn't a digit, by its one expression.
… in the docs that a term written again at once counts each time (#422)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Full-text search undercounts a term repeated back to back: 'kiwi kiwi kiwi' counts 2

1 participant