Push Spark dynamic list-index predicates into native scans #118

Problem

We need Spark to push down predicates that select a list element using an index stored in the same row:

WHERE object.list[object.idx].field = 'required'

The query selects parent objects whose indexed element matches; it does not search all elements or trim the returned list.

PR #117 adds V2 scan pushdown for array_contains and exists, but neither its translator nor the native Java expression API supports this dynamic indexed lookup. This is a follow-up to #117 and #116.

Proposed implementation

• Add a native expression such as listAt(list, index), with both arguments supplied as expressions. For each row, validate its index and gather the selected element using the list's offsets, validity, and existing array take operations. Cover the supported physical list representations.
• Expose it through Java/JNI, following existing expression ownership conventions.
• Extend the Spark V2 translator to recognize Catalyst GetArrayItem, translate the list and index expressions, and accept the surrounding struct-field access and comparison. Report both column dependencies for per-file physical-type validation.
• Preserve element-field pruning. A candidate translation is listAt(listTransform(object.list, x -> x.field), object.idx) = 'required', reusing the list layout's existing transform-based field pruning. Validate that this works through layout expression partitioning.
• Refine per-file validation so unsupported or missing sibling fields in an element struct do not disable pushdown when the predicate reads only a compatible field. PR feat(spark): push list predicates into V2 native scans #117 currently validates the complete element struct conservatively.

Correctness requirements

Match Spark's resolved GetArrayItem semantics:

• Zero-based indexing, with a different index allowed for each row.
• Correct propagation of null parent objects, lists, indices, elements, and fields.
• Correct negative and out-of-range behavior according to failOnError / ANSI mode.
• Match Spark's integral-index conversion behavior, including large values; avoid unchecked narrowing or offset arithmetic overflow.
• Retain existing comparison safeguards, including floating-point and string-collation restrictions, and fall back safely for unsupported expressions or physical types.

Returning null for an invalid index while retaining Spark's residual filter is insufficient in throwing mode: native filtering could discard the row and hide the error. Either preserve the required error semantics or leave that expression in Spark.

Acceptance criteria

• The example query is evaluated as a native Vortex filter through Spark V2 scans, with Spark residual evaluation retained.
• Regression tests cover varying and constant indices, nulls at each level, empty lists, negative/out-of-range/large indices, both ANSI modes, and physical-schema mismatches.
• Tests prove executor-side native filter construction and actual native filtering, rather than relying only on final results or scan metadata.
• I/O coverage verifies that list-of-struct layouts can avoid reading sibling element fields.
• Compile and validate across supported Spark/Scala variants.
• Measure selective-read performance against Spark residual filtering and report the result without assuming an improvement.

Performance scope

The initial goal is native filtering of parent rows plus pruning of unused element fields. Reading only the individually selected elements from storage, and list-element zone pruning, are separate optimizations. No on-disk format change appears necessary, subject to implementation validation.