Skip to content

feat(spark): add opt-in ordered reads and JMH comparison - #120

Open
ChristopherWirt wants to merge 2 commits into
developfrom
cwirt/spark-ordered-read
Open

ChristopherWirt wants to merge 2 commits into
developfrom
cwirt/spark-ordered-read

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Spark currently returns a Vortex file's decoded chunks in completion order, so callers that consume users in stored order cannot request that ordering through the connector. Add vortex.read.ordered: when true, the shared V1/V2 reader builds its scan with ordered(true). The option defaults to false, preserving existing Spark read behavior.

Related to #76, with ordering exposed as an opt-in option. The guarantee applies within each file split; it does not establish ordering across Spark partitions or files.

Changes

  • Resolve the option through the existing case-insensitive format options, including session defaults.
  • Add a JMH comparison of ordered and unordered scans over an existing file. Every column is decoded through VortexPartitionReader; schema inference is outside the timed region.
  • Document usage, benchmark commands, and the need for an optimized JNI build for performance measurements.
  • Add a 300,000-row ordering regression covering V2, V1 columnar, and V1 row readers, whole and split files, pushed filters, and mixed-case option spelling.

Representative measurements on a current bid-hour event file and featoid bucket are still pending. Enabling the option in downstream StoreFiles and removing run splitting are follow-ups contingent on those results; the check rejecting out-of-order users should remain.

Testing

  • Ran ./gradlew spotlessApply from java/, covering the root and all subprojects; reviewed the resulting diff.
  • Spark 4.0 / Scala 2.13 targeted tests passed: VortexOptionsTest, VortexPartitionReaderLifecycleTest, and VortexFileSplitTest.orderedReadsPreserveRowOrder (23 test cases; the ordering regression exercises 12 read scenarios). Temporarily removing the option forwarding made the regression fail for all three reader modes; restoring it passed.
  • On this PR branch, Spark 4.0 / Scala 2.13 JMH compilation and forked smoke runs passed for both ordered=false and ordered=true, decoding all five columns of an existing 10,565,924-byte featoid file.
  • The smoke run used debug JNI, one fork per mode, and one 100 ms warmup and measurement iteration. It validates the harness only and supports no performance conclusion.
  • Production-size comparisons, full test suites, and Spark 3.5 runs were not run.

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant