refactor(table): unify vector search scan and read APIs

[JingsongLi]

Purpose

Vector search planning, DE/PK execution, and row materialization were concentrated in vector_search_builder.rs, with separate result and execution paths for primary-key tables. This change exposes a shared Scan → Plan → Read flow, following the Java API structure, so native and externally planned searches use the same readers.

Related: #771.

Brief change log

• Separate the single-query and batch builders, DE/PK scanners and readers, query validation, result reads, and their tests into focused modules.

• Introduce common VectorScan, VectorScanPlan, VectorRead, and BatchVectorRead APIs. Readers own their configuration, and plans can be reused across queries without replanning.

• Unify DE global row IDs and PK physical positions under a snapshot-scoped SearchResult, with projected row materialization through new_read_builder(). Update Hybrid and DataFusion callers.

• Move Java bucket-split byte decoding to a standalone C API. Construct a common plan from decoded split handles and execute it through the shared C reader, preserving supplied snapshots, files, and row ranges.

• Align PK row filtering with Java rowRangesByFile: replace FileRowSelection with merged per-file ranges, coalesce residual matches during reads, and use ranges in both ANN and exact search.

Tests

• Core vector unit tests: 452 passed (cargo test -p paimon --offline --features fulltext --lib vector).
• PK baseline, batch, bucket-split, and Java-fixture integration suites: 26 passed.
• Hybrid search builder tests: 12 passed.
• C vector-search tests: 15 passed.
• DataFusion vector-search tests: 7 passed.
• Generated the C header and compiled the documented reader example with C11, -Wall -Wextra -Werror; compiled the Java serializer example with javac.
• git diff --check passed.

Regression coverage includes residual-range coalescing without bridging gaps, compact inclusive range intersection, the shared ANN mask size bound, plan reuse across single and batch readers, handle lifetimes, plan/reader context mismatches, invalid C inputs, and reading Java-planned PK splits after removing the index manifest.

API and Format

This intentionally changes Rust vector-search APIs: builders return the common SearchResult, and row reads are configured from that result. The bucket-specific Rust execution entry and C paimon_vector_search_builder_execute_read_for_bucket_splits entry are removed in favor of separate decoding, plan construction, and reading. The ordinary C paimon_vector_search_builder_execute_read remains a local convenience operation.

No persisted table or split format changes. The standalone decoder accepts Java's versioned PKVSPLIT format; Java object-stream envelopes and serialized DE index/raw splits are not supported. DE uses the common native scan/plan/read API.

Documentation

Extend docs/src/c-binding.md with the shared vector API and a complete Java-to-C example covering split serialization, table metadata, scalar residuals, Arrow ownership, cleanup, plan reuse, and distributed Top-K merging.

[leaves12138]

Reviewed 024fabb, including the shared scan/plan/read contracts, snapshot-scoped result materialization, PK residual/physical-range intersection, ANN/exact filtering, plan reuse, C handle ownership, and Hybrid/DataFusion call sites. One CI-blocking finding is recorded inline; I did not find another concrete blocking correctness issue in the reviewed paths.

Local validation passed: core vector unit tests (452), the four PK integration suites (26), Hybrid builder tests (12), C vector tests (15), DataFusion vector-search unit tests (7), executor tests (11), and cargo fmt --all -- --check. These are targeted suites, not a full workspace/platform matrix. The narrowed local Clippy run fails with the same five diagnostics as the hosted check job.

[leaves12138]

[P2] Fix the new Clippy errors so the check job can pass

The current head fails the hosted check job in Clippy, not because of an intermittent runner failure. I reproduced the same five diagnostics locally with cargo clippy -p paimon --offline --features fulltext --lib -- -D warnings:

• large_enum_variant here on VectorReadKind, and on VectorScanWork (vector_scan.rs:50) and VectorScanKind (vector_scan.rs:129). For example, this enum embeds a 1,232-byte PK reader alongside a 24-byte DE reader.
• too_many_arguments on BatchVectorRead::new (vector_read.rs:71, 8 arguments).
• needless_borrow on table: Some(&pinned_table) (de_vector_read.rs:228); pinned_table is already a reference.

Please address the enum layout warnings (for example by boxing the large variants), remove the redundant borrow, and refactor or narrowly annotate the constructor argument count. Then rerun the workspace Clippy command used by CI. These are errors under the existing -D warnings policy, so passing the runtime tests alone does not make this head pass CI.

[leaves12138]

Re-reviewed be23031. The previous Clippy blockers are resolved; no additional blocking findings in the six-file follow-up. The large enum variants are boxed, both builder paths still construct and validate the same PlanContext before creating their readers, and the redundant borrow and test casts are removed. Archive comparison confirms all other files are unchanged from the reviewed head.

Re-ran locally: core vector unit tests (452), the four PK integration suites (26), C vector tests (15), and formatting; all passed. Clippy also passes locally for all non-Python workspace targets with fulltext,vortex and -D warnings. The full local workspace attempt was blocked by the installed Python distribution lacking a shared library, so I additionally checked the hosted check job: its full-workspace, all-targets Clippy command and formatting both passed on this exact commit.

This approval does not imply that the complete hosted platform/integration matrix has finished.