Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed

- `dev.vortex:vortex-jni` 0.84.0 → 0.85.0; vortex-jni's writer no longer emits a per-zone `SUM` in the `vortex.zoned` stats table (upstream: a zone sum prunes nothing and its null-on-empty semantics were unsettled), so `ZoneReducer#sum` now falls back to a full scan for Rust-written files instead of pushing the reduction down. ([#360](https://github.com/dfa1/vortex-java/pull/360))

## [0.13.3] — 2026-08-15

Nested Parquet import, plus a cluster of real correctness bugs (three writer-side, one reader-side)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -248,10 +248,16 @@ void jniWriter_javaReader_singleChunk(@TempDir Path tmp) throws IOException {
}

@Test
void jniWriter_perZoneSum_readFromZoneMapTable(@TempDir Path tmp) throws IOException {
void jniWriter_noPerZoneSum_zoneReducerSignalsFallbackAndDecodeStillCorrect(@TempDir Path tmp)
throws IOException {
// Given — a Rust-written file large enough that the JNI writer emits a multi-zone column.
// Sum lives only in Rust's vortex.stats zone-map table (its flat writer doesn't retain it),
// so this proves the Java reader decodes that table for per-zone SUM (ADR 0013 §6 parity).
// vortex-jni 0.85.0 dropped SUM from the zone-map aggregate defaults (a zone sum prunes
// nothing, and its null-on-empty semantics were unsettled — see spiraldb/vortex#9206's
// writer comment), so zones now carry only MAX/MIN/NAN_COUNT/NULL_COUNT. This proves the
// Java reader reflects that absence faithfully (no fabricated SUM), that
// [io.github.dfa1.vortex.reader.compute.ZoneReducer#sum(String)] signals the caller to fall
// back instead of under-counting, and that decoding the column directly still yields the
// right total.
int n = 200_000;
long[] ids = new long[n];
double[] vals = new double[n];
Expand All @@ -263,15 +269,22 @@ void jniWriter_perZoneSum_readFromZoneMapTable(@TempDir Path tmp) throws IOExcep
writeJni(file, ids, vals);
long expected = (long) n * (n - 1) / 2; // Σ 0..n-1

// When — fold the per-zone SUM rows the reader surfaces from the zone-map table
// When / Then — no zone carries a SUM statistic anymore
try (var vf = VortexReader.open(file, ReadRegistry.loadAll());
var iter = vf.scan(io.github.dfa1.vortex.reader.ScanOptions.all())) {
List<ArrayStats> zones = iter.columnZoneStats("id");
assertThat(zones).isNotEmpty().allSatisfy(z -> assertThat(z.sum()).isNull());
}

// When / Then — the reducer refuses to guess and tells the caller to stream instead
try (var vf = VortexReader.open(file, ReadRegistry.loadAll())) {
Number pushedDown = new io.github.dfa1.vortex.reader.compute.ZoneReducer(vf).sum("id");
assertThat(pushedDown).isNull();
}

// Then — every zone carries a SUM (came from Rust's table, not a Java-side recompute)
// and the whole-zone fold equals the column total.
assertThat(zones).isNotEmpty().allSatisfy(z -> assertThat(z.sum()).isNotNull());
long total = zones.stream().mapToLong(z -> (Long) z.sum()).sum();
// When / Then — decoding the column the ordinary way still returns the exact total
try (var vf = VortexReader.open(file, ReadRegistry.loadAll())) {
long total = scanAll(vf).stream().flatMapToLong(c -> Arrays.stream(ids(c))).sum();
assertThat(total).isEqualTo(expected);
}
}
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@
<jazzer.version>0.30.0</jazzer.version>
<assertj.version>3.27.7</assertj.version>
<mockito.version>5.23.0</mockito.version>
<vortex-jni.version>0.84.0</vortex-jni.version>
<vortex-jni.version>0.85.0</vortex-jni.version>
<arrow.version>19.0.0</arrow.version>
<slf4j.version>2.0.18</slf4j.version>
<!-- coverage -->
Expand Down
Loading