Skip to content

Read a query's words as the text's expression does, Alphabetic letters, decimal digits and underscores (#434) - #437

Draft
jasperblues wants to merge 2 commits into
mainfrom
fix-434-word-characters
Draft

jasperblues wants to merge 2 commits into
mainfrom
fix-434-word-characters

Conversation

@jasperblues

Copy link
Copy Markdown
Member

Closes #434. Port of liberation-data/drivine4j#616 (drivine4j PR #627), as liberation-data/drivine#643 ports it. Not to merge until train 2026.11.1 is published, and not before drivine#643.

text_query.py split a full-text query into words with \W. Python's \w is what str.isalnum holds, plus _. So it kept ², ½ and ⅻ in a word, and split a Hindi word at its vowel sign, which isalnum doesn't hold.

A term's expression reads the text with [[:alnum:]_], which on Postgres, and through sqlite_patterns (Java's \p{Alnum}), is Unicode's Alphabetic plus decimal digits. So किताब never matched, and ² stayed in a word where the text, and Neo4j's analyzer, end one at it.

A query's words are now split by _gap(): not Alphabetic, Nd or _, built from sqlite_patterns.property_ranges("Alnum") and class_text. It is built on the first query rather than on import, because it reads every code point (about 85 ms).

Goldens: unchanged.

Tests

  • tests/integration/test_bm25.py, and its sync copy, on Neo4j, Postgres, Postgres indexed, SQLite and Oracle: किताब is found, and "m²x" finds "an m² x here". many's pinned score is 0.0989 for the two new seed texts.
  • tests/unit/test_text_query.py:
    • the words of किताब, bⅻb, m²x, 1½2, ٣٤ and m\²x;
    • SQLite's [[:alnum:]_] and a query's words hold the same characters.

Verification

  • Two fresh-reviewer passes. Pass 1 added the [[:alnum:]_] pin and the docs notes (docs/02-connecting.md, docs/07-schema.md); pass 2 came back clean.
  • Unit tests: 2,484 passed. ruff and pyright are clean, and the sync code is current.
  • Integration, on the test box with Oracle: the BM25 tests, their sync copy, the search tests, and the SQLite and Postgres store tests. 223 passed, and agreement with Neo4j is unchanged at 17/21.

…s, decimal digits and underscores (#434)

Port of liberation-data/drivine4j#616 (drivine4j PR #627). \W split a Hindi word at its vowel sign,
so it never matched, and kept ² in a word where the text ends one at it. The class is built from
sqlite_patterns' Alnum ranges, on the first query rather than on import.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant