GH-48344: [Python] Fix Table.from_struct_array for empty ChunkedArray - #49869
Conversation
|
|
5b68bdb to
a258e03
Compare
|
|
|
|
AlenkaF
left a comment
There was a problem hiding this comment.
Thanks for the contribution! Looks good to me, just added a nit to the test comment.
There was a problem hiding this comment.
Pull request overview
Fixes pyarrow.Table.from_struct_array to correctly round-trip empty tables when the input is a zero-chunk struct ChunkedArray by ensuring Table.from_batches receives an explicit schema.
Changes:
- Pass
schema=schema(struct_array.type.fields)when constructing aTablefrom a structChunkedArray(including the zero-chunk case). - Add a regression test covering
Table.from_struct_arraywith an empty structChunkedArray.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| python/pyarrow/table.pxi | Ensures from_struct_array supplies a schema to from_batches for chunked struct inputs, fixing the zero-chunk path. |
| python/pyarrow/tests/test_table.py | Adds a regression test for constructing a table from a zero-chunk struct ChunkedArray. |
|
|
|
@1fanwang could you take a look at the Copilot review comment? |
AlenkaF
left a comment
There was a problem hiding this comment.
Thanks for the update, looks good now!
One minor thing: could you update the description of the PR so it reflects the current change: schema=schema(struct_array.type.fields) is not used anymore but the combine_chunks() is in case of empty arrays.
The linter error should be fixed with a rebase.
…dArray Round-tripping an empty Table through to_struct_array -> from_struct_array raised "Must pass schema, or at least one RecordBatch": to_struct_array returns a ChunkedArray with zero chunks for an empty table (since apache#46355), and from_struct_array then called Table.from_batches([]) without a schema. Pass schema=schema(struct_array.type.fields) so the zero-chunk path preserves the field names and dtypes from the ChunkedArray's struct type. Mirrors the inverse fix in apache#46355. Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
3f4a298 to
e4ad179
Compare
Rationale for this change
Round-tripping an empty table through
to_struct_array()andTable.from_struct_array()raisesValueError: Must pass schema, or at least one RecordBatch.to_struct_array()returns a zero-chunk structChunkedArray, sofrom_struct_array()has no batch from which to recover the schema.For a zero-chunk input,
from_struct_array()now callscombine_chunks()andconverts the resulting empty struct array into a
RecordBatch. Nonempty inputskeep the existing path. An empty non-struct
ChunkedArraystill raises theexpected
TypeError.What changes are included in this PR?
The empty
ChunkedArraypath creates one empty batch before callingTable.from_batches. The regression coverage checks the round-trip result,schema preservation, and the invalid non-struct input.
Are these changes tested?
I ran this script against PyArrow 25.0.1 and the compiled branch:
Before:
After:
The focused regression suite also runs against the compiled branch:
Are there any user-facing changes?
Table.from_struct_array()now returns the expected empty table for azero-chunk struct
ChunkedArrayinstead of raisingValueError.GitHub Issue:
pa.Table.from_struct_arrayfails for an empty array #48344GitHub Issue:
pa.Table.from_struct_arrayfails for an empty array #48344