Skip to content

Functions: match short NGRAM query terms under column collation | pd=release-8.5 tikv=release-8.5 tidb=release-8.5 ticdc=release-8.5 - #11364

Merged
ti-chi-bot[bot] merged 5 commits into
pingcap:feature/release-8.5-ftsfrom
solotzg:codex/fts-ngram-short-query
Nov 20, 2026
Merged

ti-chi-bot[bot] merged 5 commits into
pingcap:feature/release-8.5-ftsfrom
solotzg:codex/fts-ngram-short-query

Conversation

@solotzg

@solotzg solotzg commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: ref #10233

Short NGRAM Boolean query terms can be dropped before TiFlash compares them with document tokens. A single digit such as 4 can compare equal to a document bigram containing that digit and U+2063 under a collation that ignores U+2063.

Paired with TiDB PR #73458.

What is changed and how it works?

Retain short NGRAM Boolean query terms for collation-aware matching. Keep
fixed-size document ngrams and the existing Local MATCH protocol version.
Add pushed-down DAG coverage for wrapped and unwrapped queries across
binary, general, Unicode, and 0900 collations.

Check List

Tests

  • Unit test
  • Integration test

All 41 tests in gtests_dbms --gtest_filter='TestLocalMatchAgainst.*' passed, including the DAG and short-term early-exit regressions.

Linux x86-64 SQL validation

Compared TiFlash e17844b80c (baseline) and b56cda95ff (candidate) with the same TiDB build (27b77dbfda, including TiDB #73458), PD, TiKV, and proxy. Both TiFlash binaries used the same optimized build settings.

  • Correctness. All 35 SQL cases returned the expected ordered IDs. On the 300,000-row wrapped-digit fixture, the candidate returned 56,470 rows for both the wrapped and bare quoted 4 under utf8mb4_unicode_ci and utf8mb4_0900_ai_ci, up from zero on the baseline and equal to TiDB/TiKV and bytewise LIKE. The utf8mb4_bin and utf8mb4_general_ci controls returned zero as expected. With stopwords off, ban* and +banana each matched 60,000 rows. With stopwords on, +banana returned zero. WITH PARSER NGRAM emitted no DDL warning.
  • SQL boundaries. All 62 additional cases agreed between TiDB/TiKV and TiFlash with ngram_token_size 1, 2, and 3. They cover prefixes, quoted terms, required and prohibited clauses, and phrase positions with stopwords.
  • Performance. Across two AB/BA pairs, none of the 14 queries with identical results exceeded the wall-time regression threshold (max(5% of baseline, 0.2 ms)). The tested Chinese NGRAM queries were 4–6% faster and the English word and phrase controls were 10–13% faster. Four binary/general short-query controls also stayed within the wall-time and CPU thresholds in a 21-sample-per-version-per-pair follow-up. Restoring UCA short-query matching increased wall time from about 17 ms to 35–39 ms and TiFlash CPU time from about 9–13 ms to 38–43 ms per query while restoring 56,470 matches.

Reported F4 traversal

Ran the original F4 SQL against 8,000-row synthetic fixtures using the reported schema shape. Forced TiFlash evaluated MATCH in MPP. With reads restricted to TiKV, TiDB evaluated MATCH at the root.

text_value_20 collation ngram_token_size Default plan Forced TiFlash TiKV scan + TiDB MATCH LIKE fallback
utf8mb4_bin (reported) 2 0 0 0 1,000
utf8mb4_unicode_ci 2 1,000 1,000 1,000 1,000
utf8mb4_bin 1 1,000 1,000 1,000 —

The reported binary-collation F4 case therefore remains unresolved at gram size 2. The 8,000-row run validates results and execution paths. The original 30-million-row workload was outside this run.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix missed matches for short NGRAM Boolean queries pushed down to TiFlash when the column collation equates the query term with a document ngram.

Summary by CodeRabbit

  • New Features
    • NGRAM searches can match short terms and phrases according to the selected collation. Prefixes and viable terms continue to be matched.
    • Boolean searches now handle required and prohibited clauses more consistently, including early decisions when the outcome is clear.
  • Bug Fixes
    • Impossible short NGRAM terms and phrases are rejected before document scanning where supported.
    • NULL search values return no matches, and NULL document values no longer contribute searchable text.
    • Phrase matching remains accurate when filtering changes the terms used for matching.

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 19, 2026
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

Local MATCH now analyzes NGRAM documents, queries, and prefixes in distinct modes. Query compilation uses collation metadata and checks short NGRAM phrases. Boolean evaluation tracks required-clause matches, and column access handles constant and nullable strings. Tests cover short terms and phrases across collations.

Changes: Local MATCH Boolean evaluation

Layer / File(s) Summary
Token and short-run analysis
dbms/src/Functions/FunctionsLocalMatchAgainst.cpp
Token storage no longer keeps code-point counts. NGRAM analysis distinguishes document, query, and prefix modes. Query analysis retains eligible short runs, and stopword checks search candidate substrings across each run.
Query compilation and clause matching
dbms/src/Functions/FunctionsLocalMatchAgainst.cpp
Compilation uses collation metadata, rejects impossible short phrases in specified cases, and retains unfiltered phrase verification when filtering changes phrase terms. Prefix clauses use one compiled matcher.
Column access and Boolean row evaluation
dbms/src/Functions/FunctionsLocalMatchAgainst.cpp, dbms/src/Functions/tests/gtest_local_match_against.cpp
Boolean matching tracks required-clause matches and handles prohibited clauses during traversal. Column access handles constant and nullable strings. Execution uses block-size routing thresholds, and tests cover short terms and phrases across collations.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Possibly related PRs

  • pingcap/tiflash#11345: Backports Local MATCH Boolean support to release-8.5, including the evaluation path changed here.
  • pingcap/tiflash#11295: Adds the release-8.5 Local MATCH implementation, including NGRAM query handling changed here.

Suggested reviewers: xzhangxian1008, windtalker

Merge Risk: Minimal · up to 25ba6

This change makes short NGRAM Boolean queries pushed down to TiFlash match documents that the column collation treats as equal. It keeps the existing protocol version and adds regression tests across several collations. The one open question is whether TiDB's own evaluator behaves the same way for this case. That could not be checked from this repository, and nothing found shows a problem.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: short NGRAM query terms are now matched under column collation in the Functions code. It also includes release branch tags, which are noise but do not make the title u…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the repository template and covers the problem, the change, test evidence, side-effect and documentation checkboxes, and a release note. It states the problem clearly: short NG…
Full details: Title check

Explanation

The title names the main change: short NGRAM query terms are now matched under column collation in the Functions code. It also includes release branch tags, which are noise but do not make the title unrelated. The summary matches the changeset, which retains short NGRAM terms and adds collation-aware tests.

Full details: Description check

Explanation

The description follows the repository template and covers the problem, the change, test evidence, side-effect and documentation checkboxes, and a release note. It states the problem clearly: short NGRAM Boolean query terms can be dropped before collation-aware comparison, so matches such as the digit 4 against a bigram containing U+2063 are missed. It links the paired TiDB PR and reports unit, integration, and SQL validation results. The Issue Number is written as 'ref #10233' rather than 'close #10233', which is acceptable because the issue is referenced but the template expects a closing keyword only when the issue is resolved. The F4 table states that the reported binary-collation case remains unresolved at gram size 2, which the author discloses openly. The description is thorough and mostly complete. Minor gaps: the 'Side effects' checkboxes are all unchecked even though the performance section reports a CPU and wall-time increase for the restored UCA short-query path, and the commit-message block is brief. These are non-critical for a pass.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A rabbit reads the NGRAM stream,
And finds short words that fit the scheme.
Through collations, phrases hop,
Required clauses count and stop.
Nulls pass by without a sound,
While tested matches leap around.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f9c5cf5c-358d-4c25-b002-a95669d4dc02
Commits

Reviewing files that changed from the base of the PR and between 2644190 and 25ba6f5.

Files selected for processing: 2

  • dbms/src/Functions/FunctionsLocalMatchAgainst.cpp
  • dbms/src/Functions/tests/gtest_local_match_against.cpp

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

@ti-chi-bot ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 19, 2026
Bind MATCH collation capabilities once and reject short exact clauses before scanning documents when collation equality requires equal code-point counts.
@ti-chi-bot ti-chi-bot Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 20, 2026
@solotzg
solotzg force-pushed the codex/fts-ngram-short-query branch from 25ba6f5 to b56cda9 Compare November 20, 2026 04:07
@ti-chi-bot ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 20, 2026

@windtalker windtalker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot

ti-chi-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: windtalker, xzhangxian1008

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm approved and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 20, 2026
@ti-chi-bot

ti-chi-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-11-20 09:21:39.954953695 +0000 UTC m=+59946.683265577: ☑️ agreed by xzhangxian1008.
  • 2026-11-20 09:45:52.681443535 +0000 UTC m=+61399.409755417: ☑️ agreed by windtalker.

@ti-chi-bot
ti-chi-bot Bot merged commit 00e3eeb into pingcap:feature/release-8.5-fts Nov 20, 2026
7 checks passed
@solotzg
solotzg deleted the codex/fts-ngram-short-query branch November 20, 2026 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants