Skip to content

GH-48344: [Python] Fix Table.from_struct_array for empty ChunkedArray - #49869

Merged
AlenkaF merged 3 commits into
apache:mainfrom
1fanwang:gh-48344-empty-struct-array
Aug 28, 2026
Merged

GH-48344: [Python] Fix Table.from_struct_array for empty ChunkedArray#49869
AlenkaF merged 3 commits into
apache:mainfrom
1fanwang:gh-48344-empty-struct-array

Conversation

@1fanwang

@1fanwang 1fanwang commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Round-tripping an empty table through to_struct_array() and
Table.from_struct_array() raises
ValueError: Must pass schema, or at least one RecordBatch.
to_struct_array() returns a zero-chunk struct ChunkedArray, so
from_struct_array() has no batch from which to recover the schema.

For a zero-chunk input, from_struct_array() now calls combine_chunks() and
converts the resulting empty struct array into a RecordBatch. Nonempty inputs
keep the existing path. An empty non-struct ChunkedArray still raises the
expected TypeError.

What changes are included in this PR?

The empty ChunkedArray path creates one empty batch before calling
Table.from_batches. The regression coverage checks the round-trip result,
schema preservation, and the invalid non-struct input.

Are these changes tested?

I ran this script against PyArrow 25.0.1 and the compiled branch:

python - <<'PY'
import pyarrow as pa

value = pa.chunked_array(
    [], type=pa.struct([("ints", pa.int32()), ("floats", pa.float32())])
)
print(f"pyarrow={pa.__version__}, chunks={value.num_chunks}")
try:
    result = pa.Table.from_struct_array(value)
    print(f"rows={result.num_rows}, schema={result.schema}")
except Exception as error:
    print(f"{type(error).__name__}: {error}")
PY

Before:

pyarrow=25.0.1, chunks=0
ValueError: Must pass schema, or at least one RecordBatch

After:

pyarrow=26.0.0.dev175+ge4ad179ae, chunks=0
rows=0, schema=ints: int32
floats: float

The focused regression suite also runs against the compiled branch:

$ python -m pytest python/pyarrow/tests/test_table.py -k from_struct_array -q
.......                                                                  [100%]
7 passed, 208 deselected in 0.51s

Are there any user-facing changes?

Table.from_struct_array() now returns the expected empty table for a
zero-chunk struct ChunkedArray instead of raising ValueError.

@1fanwang
1fanwang requested review from AlenkaF, raulcd and rok as code owners April 27, 2026 08:06
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #48344 has been automatically assigned in GitHub to PR creator.

@1fanwang
1fanwang force-pushed the gh-48344-empty-struct-array branch 2 times, most recently from 5b68bdb to a258e03 Compare April 27, 2026 09:01
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #48344 has been automatically assigned in GitHub to PR creator.

@1fanwang 1fanwang closed this Apr 27, 2026
@1fanwang 1fanwang reopened this Apr 27, 2026
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #48344 has been automatically assigned in GitHub to PR creator.

@AlenkaF AlenkaF left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! Looks good to me, just added a nit to the test comment.

Comment thread python/pyarrow/tests/test_table.py Outdated
@github-actions github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jun 22, 2026
Copilot AI lite review requested due to automatic review settings June 22, 2026 08:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes pyarrow.Table.from_struct_array to correctly round-trip empty tables when the input is a zero-chunk struct ChunkedArray by ensuring Table.from_batches receives an explicit schema.

Changes:

  • Pass schema=schema(struct_array.type.fields) when constructing a Table from a struct ChunkedArray (including the zero-chunk case).
  • Add a regression test covering Table.from_struct_array with an empty struct ChunkedArray.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
python/pyarrow/table.pxi Ensures from_struct_array supplies a schema to from_batches for chunked struct inputs, fixing the zero-chunk path.
python/pyarrow/tests/test_table.py Adds a regression test for constructing a table from a zero-chunk struct ChunkedArray.

Comment thread python/pyarrow/table.pxi
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #48344 has been automatically assigned in GitHub to PR creator.

@AlenkaF

AlenkaF commented Jun 23, 2026

Copy link
Copy Markdown
Member

@1fanwang could you take a look at the Copilot review comment?

Copilot AI review requested due to automatic review settings August 23, 2026 10:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@AlenkaF AlenkaF left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update, looks good now!
One minor thing: could you update the description of the PR so it reflects the current change: schema=schema(struct_array.type.fields) is not used anymore but the combine_chunks() is in case of empty arrays.

The linter error should be fixed with a rebase.

1fanwang and others added 3 commits August 27, 2026 15:13
…dArray

Round-tripping an empty Table through to_struct_array -> from_struct_array
raised "Must pass schema, or at least one RecordBatch": to_struct_array
returns a ChunkedArray with zero chunks for an empty table (since apache#46355),
and from_struct_array then called Table.from_batches([]) without a schema.

Pass schema=schema(struct_array.type.fields) so the zero-chunk path
preserves the field names and dtypes from the ChunkedArray's struct type.

Mirrors the inverse fix in apache#46355.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Copilot AI review requested due to automatic review settings August 27, 2026 19:30
@1fanwang
1fanwang force-pushed the gh-48344-empty-struct-array branch from 3f4a298 to e4ad179 Compare August 27, 2026 19:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@AlenkaF AlenkaF left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you again @1fanwang!

@AlenkaF
AlenkaF merged commit f52076c into apache:main Aug 28, 2026
40 checks passed
@AlenkaF AlenkaF removed the awaiting committer review Awaiting committer review label Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants