Repository navigation
Functions: match short NGRAM query terms under column collation | pd=release-8.5 tikv=release-8.5 tidb=release-8.5 ticdc=release-8.5 - #11364
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
Bind MATCH collation capabilities once and reject short exact clauses before scanning documents when collation equality requires equal code-point counts.
25ba6f5 to
b56cda9
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: windtalker, xzhangxian1008 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
00e3eeb
into
pingcap:feature/release-8.5-fts
What problem does this PR solve?
Issue Number: ref #10233
Short NGRAM Boolean query terms can be dropped before TiFlash compares them with document tokens. A single digit such as
4can compare equal to a document bigram containing that digit and U+2063 under a collation that ignores U+2063.Paired with TiDB PR #73458.
What is changed and how it works?
Check List
Tests
All 41 tests in
gtests_dbms --gtest_filter='TestLocalMatchAgainst.*'passed, including the DAG and short-term early-exit regressions.Linux x86-64 SQL validation
Compared TiFlash
e17844b80c(baseline) andb56cda95ff(candidate) with the same TiDB build (27b77dbfda, including TiDB #73458), PD, TiKV, and proxy. Both TiFlash binaries used the same optimized build settings.4underutf8mb4_unicode_ciandutf8mb4_0900_ai_ci, up from zero on the baseline and equal to TiDB/TiKV and bytewiseLIKE. Theutf8mb4_binandutf8mb4_general_cicontrols returned zero as expected. With stopwords off,ban*and+bananaeach matched 60,000 rows. With stopwords on,+bananareturned zero.WITH PARSER NGRAMemitted no DDL warning.ngram_token_size1, 2, and 3. They cover prefixes, quoted terms, required and prohibited clauses, and phrase positions with stopwords.max(5% of baseline, 0.2 ms)). The tested Chinese NGRAM queries were 4–6% faster and the English word and phrase controls were 10–13% faster. Four binary/general short-query controls also stayed within the wall-time and CPU thresholds in a 21-sample-per-version-per-pair follow-up. Restoring UCA short-query matching increased wall time from about 17 ms to 35–39 ms and TiFlash CPU time from about 9–13 ms to 38–43 ms per query while restoring 56,470 matches.Reported F4 traversal
Ran the original F4 SQL against 8,000-row synthetic fixtures using the reported schema shape. Forced TiFlash evaluated
MATCHin MPP. With reads restricted to TiKV, TiDB evaluatedMATCHat the root.text_value_20collationngram_token_sizeMATCHutf8mb4_bin(reported)utf8mb4_unicode_ciutf8mb4_binThe reported binary-collation F4 case therefore remains unresolved at gram size 2. The 8,000-row run validates results and execution paths. The original 30-million-row workload was outside this run.
Side effects
Documentation
Release note
Summary by CodeRabbit