Repository navigation
Count a term written again at once each time, not every other time, on the SQL stores (#422) - #431
Draft
jasperblues wants to merge 3 commits into
Draft
jasperblues wants to merge 3 commits into
jasperblues wants to merge 3 commits into
Conversation
…n Postgres, SQLite and Oracle (#422) Port of liberation-data/drivine4j#607 (drivine4j PR #619), with the statements of liberation-data/drivine#629 byte for byte. Postgres and SQLite hold to the character after a term without taking it. Oracle counts the term ending where its last word does, less those a word goes on from; a phrase whose occurrences may begin inside one another in the text with its gaps doubled; and a prefix term, or one holding a number that isn't a digit, by its one expression.
… in the docs that a term written again at once counts each time (#422)
…e counts each time (#422)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #422. Port of liberation-data/drivine4j#607 (drivine4j PR #619), with the statements of liberation-data/drivine#637 byte for byte. Not to merge until train 2026.11.1 is cut, and not before drivine#637. Once #637 merges, re-record
tests/golden/VERSIONwithscripts/sync_golden.py. The statements already match its branch byte for byte.On Postgres, SQLite and Oracle without Oracle Text, a full-text term written again at once was counted every other time: "kiwi kiwi kiwi" counted 2. Since BM25 (#423), that undercount reaches the ranking.
How each store counts now
This follows drivine4j#619 and drivine#637 (
drivine/backend/sql/text_query.py:looking_ahead,counted,SEPARATOR).Postgres and SQLite: the term's trailing boundary is a lookahead,
(?=$|[^[:alnum:]_]).Oracle, in
oracle_search._oracle_count, by one of three ways:[[:alnum:]_].A word is letters (
isalpha), decimal digits (isdecimal) and_, which is\p{L},\p{Nd}and_, as in Java and TypeScript.Tests
tests/integration/test_bm25.py, and its sync copy, on Neo4j, Postgres, Postgres indexed, SQLite and Oracle:+"la la"finds "la la land";+"tapirs tapir"finds "tapirs tapirs tapir";many's pinned score is now 0.1359.tests/unit/test_text_query_counting.py: each of drivine4j's cases counted both ways throughPatterns, with exact counts.tests/unit/test_bm25.py: the SQL and bindings for each of Oracle's three ways.*.matchingstatements, copied from drivine#637. Byte-identical.Verification