Skip to content

Merge upstream develop into QC develop (2026-11-13) - #119

Merged
ChristopherWirt merged 28 commits into
developfrom
cwirt/merge-upstream-develop-2026-11-13
Nov 13, 2026
Merged

ChristopherWirt merged 28 commits into
developfrom
cwirt/merge-upstream-develop-2026-11-13

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

QC develop is missing 26 upstream commits. Merge vortex-data/vortex:develop at d81bd4ce25088e1d15eac04486442482b88fc9ed into QC develop at 97b3669a9cd77d878d28c1bacae8c513f5f57d33.

Use a merge commit to preserve upstream ancestry for future syncs.

Changes

  • Import shared primitive comparison execution, borrowed array-slot accessors, integer statistics improvements, boolean compression, and corrupt-metadata handling.
  • Bring in compact Java/Spark writers, Spark 4.3 compatibility, the Java quickstart, and benchmark updates (#11406, #11405, #11416). Compact writes retain QC's list-layout option, writer memory controls, metadata, and session handling.
  • Reconcile the upstream read-resource cleanup and soft split-boundary ports with their QC implementations (#11367, #11128). Retain QC's reachability fences, read metrics, nested list pruning, and regression coverage alongside upstream's revised lifecycle and split tests.
  • Adopt upstream's constant-byte detachment implementation while retaining QC's buffer-release regressions.

Testing

  • Ran pinned nightly-2026-09-10 Rust formatting for all 24 affected crates and full root/subproject ./gradlew spotlessApply; reviewed and included the formatter changes.
  • git diff --check passed. No unresolved merge entries or conflict markers remain; both fetched remote tips are parents of the signed-off merge commit.
  • Imported upstream history and metadata are preserved. Upstream commit b6cc108439 has no sign-off trailer.
  • Restored the test-only SegmentFuture import lost during conflict resolution. cargo check -p vortex-layout --tests --locked, pinned Rust formatting, and git diff --check passed.
  • Tests were not executed; full workspace builds, lint, and benchmarks were not run locally.

ump45nose and others added 28 commits November 11, 2026 10:47
…10852)

## Summary

- Tracking Issue: vortex-data#10756

Corrupt or crafted metadata can make a file read panic instead of
returning a Serde error. vortex-data#10760 covers the ALP-RD side of that issue.
This PR covers the RLE and RunEnd paths called out there:

- RLE `deserialize` called `dtype.as_ptype()`, which panics when the
file dtype is not primitive. It now uses `PType::try_from(dtype)?`.
- RunEnd `deserialize` used `vortex_expect` on `usize::try_from` for
`num_runs` and `offset`, which panics on 32-bit targets when the stored
value exceeds `usize::MAX`. Both conversions now return a Serde error.

Partially addresses vortex-data#10756.

Signed-off-by: Vincent <1729065730@qq.com>
…truct have (vortex-data#11128)

## Summary

- Tracking Issue: vortex-data#11123

**Problem:** A scan splits a file wherever any column's chunk ends, and
a split decodes the whole chunk of every column it overlaps. When a
struct's fields chunk at unrelated rows, as list elements and
dictionary-encoded columns of a wide nested file do, the union of their
boundaries leaves spans of a few dozen rows. Successive splits then each
decode the same chunks of the other fields. On a 180 MB list-layout
file, a full scan held 3.6 GiB of native memory.

**Fix:** The struct reader marks boundaries that only some of its fields
have as *soft*. `SplitBy::LayoutSubSplitting` merges spans that end at a
soft boundary until they reach 8,192 rows. Boundaries that every field
has are never merged, and neither are boundaries from layouts above the
struct, so files whose columns chunk alike split as before.

**Result:** full scan of one file, with Arrow conversion as JNI does it.
Live native bytes come from a counting allocator:

| | Splits | Peak live | Time | Bytes requested |
|---|---|---|---|---|
| List layout, 180 MB, before | 139 (smallest 70 rows) | 3.57 GiB | 1.7
s | 6.2 GB |
| List layout, 180 MB, after | 60 (smallest 8,192 rows) | **1.10 GiB** |
**0.6 s** | 0.99 GB |
| Default layout, 111 MB, before | 59 | 224 MiB | 0.2 s | |
| Default layout, 111 MB, after | 48 | 224 MiB | 0.2 s | |

## Changes

- **`RowSplits` gains soft marks.** Each struct field registers into its
own `RowSplits`, and the struct takes their union. Any boundary that not
every field has is marked soft.
- The shared-boundary intersection is computed once. On the existing
256-column split-collection benchmark, the warm median falls from about
338 ms to 3 ms.
- The struct's validity child is classified like any field, so a
nullable struct merges the same way at any depth.
  - Soft marks inherited from nested structs are kept.
  - A single-field struct without validity registers exactly as before.
- `SplitBy::Layout` collects positions only
(`BoundaryMode::PositionsOnly`), so it skips the classification.
- **`SplitBy::LayoutSubSplitting`** merges spans ending at a soft
boundary until they reach `DEFAULT_MIN_SPLIT_ROWS` (8,192, the writer's
row block), or `max_rows` if that is smaller. It never merges a span
past `max_rows`. The merge runs before the existing sub-division of wide
spans.

---------

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
## Summary

vortex.dev has a new FAQ page, https://vortex.dev/faq, from
vortex-data/vortex.dev#218. This PR links it from the docs: in the
footer before Charter, as the vortex.dev footer has it, in the project
page's links, and in the docs' llms.txt beside the homepage.

## Changes

`make -C docs html` builds with no warnings, and the built footer,
project page, and llms.txt each carry the link.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Will Manning <will@willmanning.io>
…ata#11367)

## Summary

Fixes vortex-data#11364.

Closing a Spark reader currently leaves its Java-owned native read
handles and file caches alive until Cleaner processing. Make that
release explicit and idempotent, while preserving the resources still
needed by derived scans, returned partitions and Arrow batches. Ports
the lifecycle fix from [QC
#103](#103).

## Changes

- Make `DataSource` and `Scan` implement `AutoCloseable`, retaining
Cleaner fallback. Keep native calls synchronized with explicit close;
fence scan options so their expressions remain alive across JNI.
- Release partitions prefetched by `hasNext()` but never returned.
- Let Spark's owning wrappers close at EOF. Reader and footer cleanup
attempts every close in ownership order and preserves the original
failure.
- Release native partition ownership when Arrow stream allocation or
import fails.
- Reject reads on closed handles with `IllegalStateException`. An open,
exhausted scan still returns `false` from `hasNext()` and throws
`NoSuchElementException` from `next()`.

---------

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
…data#10346)

Computes the integer compression statistics in one hand-written pass per
need. Only `vortex-compressor/src/stats/integer.rs` changes, and every
statistic has the same value as before.

- Validity is read one `u64` per chunk of 64 values. For the runs, nulls
are filled with the previous valid value. When only the bounds are
needed, values of at most 32 bits read nulls as the identity instead,
with a branch-free select.
- Min and max are folded with lane accumulators in the same pass as the
runs. This replaces the separate `min_max` aggregate. The bounds and the
null count are cached on the array, and bounds already cached there are
reused.
- The distinct counter (none, dense or hashed) is chosen once, and the
pass is monomorphized on it. When min equals max, counting is skipped.
- The lane count depends on the type width and the instruction set.
Without AVX2, 32-bit values use 32 lanes.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary

Filtering on decimals took significant time, noticed when profiling.

## Changes

Call the kernel in a way that does not sneakily deoptimize it, and split
the kernel to avoid branching on Options.

## Perf
Disassembly shows the code vectorizes now. Benched on M4 Pro:

| | base | PR | change |
|---|---:|---:|---:|
| `between_unpack::<i64>` per query | 740 µs | 80 µs | −89% |
| per value (~600K values) | 1.25 ns | 0.14 ns | -89% |
| whole query | 2436 µs | 1849 µs | −24% |
| `between(x,400,401)`, 1 partition | 2.43 ms | 1.80 ms | −26% |
| `between(x,400,401)`, 14 partitions | 1.44 ms | 1.36 ms | −6% |

Signed-off-by: Michal Piatkowski <291740709+MassivePizza@users.noreply.github.com>
The `vortex-file` `v2::file_stats_reader` tests fail intermittently. For
example, `dynamic_proofs_refresh_before_reads` failed in a full
`vortex-file` lib run while checking vortex-data#11332.

Each test called `SESSION.clone().with_handle(handle)` on one static
session. Clones of a `VortexSession` share their state, and
`with_handle` stores the runtime handle in that shared state. Tests
running at the same time therefore overwrote each other's handle, and a
test could drive its reads with another test's runtime, including one
that had already shut down. Some readers in these tests were also built
from the static `SESSION` directly, not from the test's own session.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
GetItem, Pack and Select serialize field names through `Display`, which
escapes quotes, backslashes and control characters. Their decoders treat
those strings as raw names. After a round trip, a valid expression can
select a different field or produce a different struct schema.

Write the raw string in all four affected metadata fields. Keep the
existing protobuf schema and raw-name decoders. Regression tests execute
restored expressions against distinct raw and escaped-name fields,
covering quotes, backslashes, control characters, NUL and Unicode;
Select covers both inclusion and exclusion.

Validation: all three regressions failed against the original
serializers and pass with this change, using `cargo +ohm test --locked
-p vortex-array --lib test_roundtrip_preserves_field_names`. Scoped
formatting and whitespace checks pass. This branch is based directly on
Spiral's pinned revision so its dependency update includes only this
fix.

Previously escaped metadata cannot be unambiguously repaired: unescaping
in the reader would change legitimate literal-backslash field names.
This corrects newly serialized expressions without claiming
compatibility for ambiguous old bytes.

Refs spiraldb/spiraldb#10053; prerequisite to spiraldb/spiraldb#9986.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
…urce buffer (vortex-data#11382)

- Tracking Issue: vortex-data#10599

`compress_constant` builds the constant from `execute_scalar`, which for
a `VarBinView` is a zero-copy slice of the chunk's data buffer, or of
its views buffer for a value of at most 12 bytes. The constant is
accepted because its `nbytes` is tiny, but it keeps the whole chunk
alive, and `BufferedStrategy` weighs it by that tiny `nbytes` and
buffers it until the end of the file. Writer memory grows with the row
count.

`compress_constant` now copies the string and binary bytes of the
scalar, including the elements of a fixed-size list, into a fresh
buffer: one value per chunk. `scalar_at` stays zero-copy and
`BufferedStrategy` is unchanged.

---------

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
…ex-data#11158)

## Summary

Sparse and run-heavy boolean arrays remain bitmaps, and validity masks
embedded in leaf and container arrays are not consistently offered to
the compressor. Wide nullable schemas therefore retain repeated raw
masks.

Add boolean compression strategies and recursively compress embedded
validity, preserving logical values and dtypes.

## Changes

- Add run-end and sparse strategies for non-nullable booleans, reusing
existing encodings.
- Compress validity in primitive, boolean, decimal, and variable-width
leaves, plus struct, list, list-view, and fixed-size-list containers.
- Register the strategies in the default compressor and exclude them
from the CUDA preset.
- Add round-trip and representation coverage for clustered/scattered
booleans, both polarities, sliced inputs, leaf validity, and container
validity.

The strategies and recursive traversal are one change: the strategies
provide the encodings, and the traversal makes embedded masks eligible
for them.

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Martin Prammer <martin@spiraldb.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
…vortex-data#11404)

statpopgen's builder appended each sample's FORMAT fields as noodles
yielded them. A sample may drop trailing fields (a no-call is often just
`./.`), so on rows with no-calls `DP`, `GQ`, `AD` and the other
per-sample lists came out shorter than `GT`, and entry j stopped being
sample j. The builder now walks the record's FORMAT keys and takes a
dropped field as null.

None of the statpopgen SQL queries read these fields, but the statpopgen
compression benchmark from vortex-data#11043 compresses all of them, so its numbers
so far come from the misaligned lists. Cached statpopgen data has to be
regenerated to pick up the fix.

Signed-off-by: Martin Prammer <martin@spiraldb.com>
The Spark 4.0 module's tests run against Spark 4.1 and 4.2 to catch
binary breaks, as in vortex-data#10792, but not against Spark 4.3 (on Maven Central
since 2026-10-30). A break on 4.3 would ship unnoticed, as the 4.2 read
failure (vortex-data#10791) did.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
…x-data#11406)

The JNI writer always uses the default write strategy, so Java and Spark
users cannot select the compact one that Python
(`VortexWriteOptions.compact()`) and `vx convert --strategy compact`
already offer. vortex-data#11374 measures 169.95 MB default vs 106.52 MB compact
(37% smaller) on 3M rows.

Closes vortex-data#11374. Covers the compression part of vortex-data#7725.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Prevent dictionary cast and scalar-function rewrites from failing on
unreferenced values, with regression coverage for both cases.

---------

Signed-off-by: blaginin <github@blaginin.me>
Co-authored-by: Claude Mythos 5.1 <noreply@anthropic.com>
vortex-data#11310)

This changes is meant to catch and apply interleavings of execute and
parent rules that otherwise have no chance to interact. Practical
example of this is `Between(Slice(RunEnd))` combination. RunEnd has
optimisations for both Slice and Between, however, Slice is an execute
rule (since it pushes the slice to the values and needs to do binary
search on runs) and between is an reduce rule. After Slice runs and
reduces to `Between(RunEnd)` we never apply the reduce parent rule that
pushes `Between` to the values of RunEnd.

With this change we try to apply rule after execute_parent is done.

---------

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Exercise lowercase label conjunctions and equivalent ILIKE filters in
the PolarSignals benchmark.

Signed-off-by: Alfonso Subiotto <alfonso.subiotto@dash0.com>
## Summary

Stacked on vortex-data#11310. Until that merges, this PR shows only the last
commit.

`Lower` and `Strpos` decode their inputs inside `execute`, in a nested
executor. When the input is a slice or a mask over a run-end or chunked
array, the function gets a canonical array and runs once per row, even
when the encoding could have pushed it into its values. vortex-data#11310 lets the
executor retry parent-reduce rules once a child is rewritten, but only
for functions that hand their inputs back to the executor through
`execute_step`. This continues the `execute_step` migration from vortex-data#11100
and vortex-data#11154 for `Lower` and `Strpos`.

## Changes

- `Lower::execute_step` and `Strpos::execute_step` request their inputs
with `require_input!(args, i => AnyColumnar)`, as `Like` and `Between`
do, then call `execute`.
- Tests: `lower`'s `preserves_runs_and_dictionary` gets a sliced case,
and `strpos` gets `preserves_runs_of_a_slice`. Both check that a
function over a sliced run-end input stays run-end encoded. Both fail
without this change, and both fail without vortex-data#11310.

Checks:
- `cargo test -p vortex-array` passes.
- `cargo clippy -p vortex-array --all-targets --all-features -- -D
warnings` passes.
- `cargo +nightly-2026-09-10 fmt -p vortex-array --check` passes.

Benchmark numbers will follow. No existing benchmark covers scalar
functions over sliced or masked encoded inputs.

This change was written with help from an AI coding agent and I haven't
reviewed it yet.

---------

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Signed-off-by: Alfonso Subiotto <alfonso.subiotto@dash0.com>
Co-authored-by: Robert Kruszewski <robert@spiraldb.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: Robert Kruszewski <github@robertk.io>
…data#11397)

- Tracking Issue: vortex-data#11392

`Literal::serialize` and `PreparedSet`'s serializer converted their
scalar through the infallible `From<&Scalar> for pb::Scalar`, which
calls `vortex_expect` on the fallible dtype serializer. An extension
dtype whose metadata serializer returns an error (for example a plugin
extension) therefore panicked through the scalar-function serializers'
`VortexResult` boundary instead of returning the error.

## Changes

- Add a fallible `Scalar::to_proto(&self) -> VortexResult<pb::Scalar>`
next to the existing `from_proto`. The `From<&Scalar>` impl now
delegates to it, so existing callers keep their behaviour.
- Use `Scalar::to_proto` in `Literal::serialize` and
`PreparedSet::serialize`, so the extension metadata error is propagated.
- Add regression tests in `vortex-array/src/scalar/proto.rs` with a test
extension type whose `serialize_metadata` always fails, checking that
`Scalar::to_proto`, `Literal::serialize`, and `PreparedSet::serialize`
all return the error rather than panicking.

Signed-off-by: Vincent <1729065730@qq.com>
…ortex-data#11395)

## Summary

- Tracking Issue: vortex-data#11393

`deserialize_row_encoding_options` reads a `u32` field count from the
metadata and computed the expected length as `4 + 2 * n` before
validating the input length. On 32-bit targets a count like
`0x8000_0000` overflows `usize` there, which panics in checked builds
instead of returning an error.

## Changes

- Compute the expected metadata length with `checked_mul` /
`checked_add` and bail with an error if it overflows, so oversized
counts are rejected the same way as other malformed metadata.

Signed-off-by: Vincent <1729065730@qq.com>
Add `tinystories/text` to the string-bench input catalog: the first
33,121 rows of the TinyStories train split (about 28 MiB). This is the
size of one file that the NeMo Curator text IO benchmark writes.

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
…ex-data#11418)

The generated {Encoding}ArraySlotsExt accessors took &self, so on an
ArrayView<'a, V> the returned slot borrowed from the local view rather
than from the array it points at. A slot read from a temporary view,
such as array.as_::<Delta>().deltas(), could not outlive that view.

Add BorrowArraySlots<'a, V>, implemented for &'a T where T:
TypedArrayRef<V>
and for ArrayView<'a, V>, which hands out the slots borrowed for 'a. The
generated ext trait becomes {Encoding}ArraySlotsExt<'a>,
blanket-implemented over it, with accessors that take self by value and
return &'a ArrayRef, Option<&'a ArrayRef>, SlotSlice<'a>, or
{Struct}View<'a>. Owned arrays keep borrowing from the array; views now
return references with lifetime 'a. The trait names are unchanged, so
imports stay the same. Encoding ext traits take TypedArrayRef<V> as
their supertrait instead of the slots trait.

Because by-value accessors take precedence over ArrayView's inherent
&self methods, a slot accessor named validity would hide
ArrayView::validity. The macro now rejects accessor names that clash
with ArrayView methods and accepts #[slot(N, accessor = name)] to rename
only the accessor. The 15 validity slots use accessor = validity_slot,
keeping their field, index constant, and displayed slot name.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the "under construction" Java quickstart
(`docs/getting-started/java.md`) with a full guide to `vortex-jni`, and
updates the related Spark and developer docs.

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
…essor (vortex-data#11423)

Follow-up to vortex-data#11418. The `#[array_slots]` accessors now take `ArrayView`
by value, so a slot field named `validity` would generate an accessor
that hides `ArrayView::validity`. vortex-data#11418 worked around this with a
`#[slot(N, accessor = name)]` option that renamed only the accessor.
That left 15 slot structs with a field called `validity` but an accessor
called `validity_slot`.

This drops the accessor option and renames the fields instead. The
field, view field, accessor and index constant now share one name,
matching Delta and BitPacked, which already use `validity_child`.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Read take indices through their unsigned reinterpretation, matching the
varbin and list take kernels, so signed index types no longer produce
separate copies of the kernels.

The sequence take kernel also evaluates in the unsigned counterpart of
the sequence ptype, since wrapping arithmetic produces the same bits in
either signedness, and reinterprets the result back. This cuts it from
64 instantiations to 16. The run-end take kernel drops from 8 to 4 index
widths in TakeExecute.

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
…ta#10676)

Tracking Issue: vortex-data#9128

## Summary

Moves primitive comparisons onto the shared RowFn executor from vortex-data#10675,
so comparison code only selects the predicate while the framework
handles decoding, constants, validity, and Boolean output.

## Changes

Preserves the public `Binary` identity and floating-point total
ordering. Uses borrowed operands and dtypes, and moves a constant left
operand to the right with the swapped operator. Relies on the existing
`primitive_comparisons_across_bitmap_words` test for all six swapped
operators and keeps the existing comparison benchmarks.

Based on vortex-data#10675. Keeps the wide-type, sliced-input, bitmap-boundary,
NaN, and signed-zero tests added upstream in vortex-data#10843.

## Performance

CodSpeed compares this PR with vortex-data#10675 (`b0a64dd`). Nine primitive
comparison benchmarks are 12% to 31% faster in walltime, led by
`compare_u64_avx2` and `compare_float_avx2` at 31% and 28%, and no
comparison benchmark regresses.

These gains depend on vortex-data#10675. vortex-data#11220 measured the same two commits
directly on `develop`. Without vortex-data#10675, comparisons on small arrays were
10% to 16% slower than `develop` in CPU simulation, and only two
walltime gains remained, at 16% and 18%. The vortex-data#10675 description has the
full comparison.

The baseline already includes the private NEON packing kernel from
vortex-data#10843, which this PR replaces with RowFn execution. So the gains above
are measured against that improved baseline, and the old packing gains
aren't credited to this PR.

<details>
<summary>CodSpeed comparison benchmarks against vortex-data#10675</summary>

CodSpeed marks the walltime runners as unreliable, but all nine cases
move in the same direction.

| Benchmark | Faster by |
| --- | ---: |
| `compare_u64_avx2` | 30.8% |
| `compare_float_avx2` | 28.2% |
| `lt_i64_nullable_avx2` | 18.4% |
| `compare_int_constant_neon` | 16.2% |
| `compare_int_eq_avx2` | 15.4% |
| `compare_f32_avx2` | 15.4% |
| `compare_int_constant_avx2` | 15.3% |
| `compare_int_avx2` | 13.5% |
| `compare_int_nullable_avx2` | 12.5% |

</details>

<details>
<summary>Historical EC2 accumulated-stack measurements</summary>

These runs measured an earlier accumulated stack, not this layer in
isolation or the current head. They showed gains on some configurations
and regressions on others.

Ratios are the median of 21 paired candidate/baseline elapsed-time
ratios. Below 1 is faster. This compares the earlier complete migration,
M0 `ddf8a16df6b7d7c20cfd860e5f23438d97ca4238`, with D
`0291778db36809f77ce0219bf9850506d3cba3b6`. That baseline predates
vortex-data#10843. The array-pair cases use all-valid nullable inputs. The addition
case is an arithmetic control whose algorithm was unchanged.

| Workload | C7g NEON | C6a AVX2 | C7i AVX2 build | C7i AVX-512 build |
| --- | ---: | ---: | ---: | ---: |
| `f32` array-pair Gte, 8,192 rows | 1.014 | 1.040 | 0.873 | 1.036 |
| `i64` array-pair Gte, 8,192 rows | 0.900 | 1.139 | 0.929 | 0.943 |
| Nullable `i64` add, 32,768 rows | 1.002 | 1.123 | 1.154 | 1.004 |

For example, the old stack took about 10% less time for the ARM `i64`
comparison, but about 14% more for the same case on AVX2-only C6a. C6a
nullable `i64` addition took about 12% more time. These whole-executable
measurements do not establish that the comparison algorithm caused
movement in arithmetic controls.

In the four-label matrix, D ran before M0 in 11 blocks and after it in
ten. The experimental one-pass decoder variants are excluded from this
table.

EC2 C7g (Neoverse V1), C6a (EPYC 7R13), and C7i (Xeon Platinum 8488C).
Rust 1.99.1 / LLVM 23.1.1, bench profile with 16 CGUs and no LTO,
MiMalloc, CPU 2 pinned. The same 455-case public-path microbenchmark
suite ran serially across hosts, with 21 paired blocks, 10 ms per case,
A/A calibration, and a separate control executable. These are
microbenchmarks, not query results. C7i can dispatch to AVX-512 at
runtime even in its AVX2 build.

Saved evidence is in the local investigation archive
`row-fn-evaluation-20261026`: `manifest.json`, `*-matrix-*-medians.csv`,
`*-matrix-*-summary.json`, and `results/<host>/results/matrix-*.json`.

</details>

<details>
<summary>Confirmed independent packing gains, now in develop via
vortex-data#10843</summary>

The ARM-only private-kernel change reduced long array-pair cohort time
on Graviton3 as follows. Each cohort contains 32 cases, including
partially valid inputs and lengths around 16K.

| Cohort | Confirmation elapsed-time ratio | Paired 95% interval | Time
reduction |
| --- | ---: | --- | ---: |
| `i64` | 0.846 | 0.843 to 0.856 | 15.4% |
| `u64` | 0.860 | 0.858 to 0.864 | 14.0% |
| `f64` | 0.882 | 0.880 to 0.886 | 11.8% |

Baseline `a59294e35e7109fac894a7a38a68f42db0c3c967`, candidate
`f018aaeef739145e1bc50079a441c5d9fd53404a`. The cohort uses a geometric
mean of case ratios within each block, then the median across blocks.
Two 21-block sessions used complementary AB/BA orders, the same full
455-case suite, and 1,000 single-iteration samples per case. The initial
fixed-order source comparisons were discarded. The AVX2 builds had
identical `.text` bytes, so no x86 speedup was attributed to this
change. See vortex-data#10843 for the harness and full methodology.

These gains required neither this PR nor vortex-data#10675.

</details>

On `e724610`, CI passed all-target, all-feature Clippy with warnings
denied and the Linux, CPU feature, sanitizer, and Miri test jobs.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt
ChristopherWirt merged commit fbc5efb into develop Nov 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.