[BubbleCal]

Summary

Fix two independent recall failures seen on main:

• In streaming coreset training, a singleton can retain a large compression loss and reach the top of the split heap while other clusters are still splittable. Finalize that singleton and continue; stop only when the heap's remaining clusters are finalized. This prevents premature termination followed by duplicate-centroid padding. Extend the existing regression test with lossy summaries and exhausted splits.
• Give test_legacy_ivf_pq_cosine_multivec_smoke eight PQ sub-vectors instead of four, reducing quantization error with its small training sample. Keep its data, query, training sample rate, iteration count, and recall >= 0.5 assertion unchanged.

Failing CI: Float64 streaming coreset recall, legacy multivector PQ recall.

Related: #9346 proposes fallback bisection and rejecting unattainable partition counts. This PR fixes singleton termination in the existing split loop and preserves the current exhausted-split behavior.

Validation

• The added lossy/exhausted regression cases fail before the fix: 16 distinct centroids instead of 32. All three cases pass afterward.
• 15 weighted tests, 28 streaming tests, and the final legacy multivector smoke test pass.
• cargo fmt --all --check
• cargo clippy --locked --all --tests --benches -- -D warnings
• git diff --check; commit hooks pass.

Local recall checks were measured on fix commit 271c19a82f6b8a7c2cd9574aca3a9bb8198f664c against 77b7fd4250e248cce207c54389b7f7065f5252d9, before syncing main, on Apple M3 Max (arm64), Rust 1.98.1, default features and test profile. Each version uses the same seeded fixture; training runs use normal OS randomness and are independent.

Scenario / metric	Baseline	Fix commit 271c19a82
Streaming L2: recall@5 below 0.5	2 / 500 runs	0 / 500 runs
Streaming cosine: recall@5 below 0.5	2 / 500 runs	0 / 500 runs
Streaming L2: distinct centroids	107–176	257
Streaming cosine: distinct centroids	80–140	257
Legacy multivector PQ: mean recall@100	0.57940	0.76845
Legacy multivector PQ: minimum recall@100	0.50	0.67

Streaming: 600 Float64 vectors, four dimensions, two fragments, 257 partitions, sample rates 2/1, one training iteration, minimum 16 probes, query row 321. All-partition recall is 1.0 in every run before and after.

Multivector PQ: 200 runs per version, 512 rows with three 32-dimensional vectors each, one fully probed partition, query row 0, 8-bit codebooks, sample rate 1, two iterations. The Linux CI failure was 0.49; the local baseline did not fall below 0.5. These are fixture-specific correctness observations.

CI on 271c19a82 passed Rust tests on all platforms, Clippy, Python and JNI checks. Its only failed check was the main-branch dependency advisory RUSTSEC-2026-0385. Syncing main incorporates the already-merged handling in #10269. The merged tree was rechecked with formatting, full workspace Clippy, and all 44 focused tests above; new CI results are pending.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[lance-gatekeeper]

✅ Gate recommendation: approve.

No blocking findings. The lossy and exhausted-split regressions fail with the former singleton exit and pass on this revision; streaming recall and legacy multivector PQ checks also pass.