Convert nested result columns without numpy float conversion - #960
Open
aminghadersohi wants to merge 2 commits into
Open
aminghadersohi wants to merge 2 commits into
aminghadersohi wants to merge 2 commits into
Conversation
With pandas enabled, ARRAY/MAP/STRUCT values went through numpy: an ARRAY<BIGINT> containing a NULL came back as float64, so values beyond 2**53 lost precision and NULL became NaN. Take nested columns directly from Arrow (to_pylist), as the disable_pandas path already does. Signed-off-by: Amin Ghadersohi <amin.ghadersohi@gmail.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved API compatibility and nested-value conversion performance concerns remain.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Updates Arrow result conversion to preserve exact nested values without NumPy float conversion.
Changes:
- Converts nested columns with
to_pylist(). - Adds regression coverage for nullable large
BIGINTarrays. - Leaves scalar pandas conversion unchanged.
| File | Description |
|---|---|
src/databricks/sql/result_set.py |
Preserves nested Arrow values during conversion. |
tests/unit/test_pandas_compatibility.py |
Tests exact nullable nested integer preservation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Nested columns are converted from Arrow directly, so they are no longer converted through pandas and then replaced. ARRAY values stay numpy.ndarray, the documented type: native dtype when the elements are float, or integer/boolean without NULLs; object dtype with exact int and None when integer elements contain NULL (previously float64 with NaN). MAP stays a list of tuples and STRUCT a dict, with the same conversion applied to their nested values. Signed-off-by: Amin Ghadersohi <amin.ghadersohi@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Description
With pandas enabled (the default),
ResultSet._convert_arrow_tableconverted every column throughto_pandas()andto_numpy(). For nested columns anARRAY<BIGINT>containing a NULL became a float64 array. Observed on a SQL warehouse with 4.4.0 and 4.6.0:array(9007199254740993, -9223372036854775808, NULL)read back as[9007199254740992.0, -9.223372036854776e+18, nan].Change:
_use_arrow_native_complex_typesdocumentation: ARRAY is anumpy.ndarray, MAP a list of(key, value)tuples, STRUCT a dict, at every nesting level.int64). An integer array with a NULL is now an objectndarrayof exactint/Noneinstead of float64 with NaN. The same applies to integer values nested inside MAP and STRUCT values.# Unreleased.How is this tested?
tests/unit/test_pandas_compatibility.py:test_list_of_bigint_with_null_is_exact(fails onmain), plus tests for native dtypes of NULL-free arrays, nested ARRAY/MAP/STRUCT values, and a table of only nested columns split across chunks. The original fix was verified live against a SQL warehouse (Core select of ARRAY/DECIMAL(38,18)/BOOLEAN with NULLs, exact values); the follow-up commit restoringnumpy.ndarrayfor arrays has not yet been run against a warehouse or through the unit suite locally, so CI is the check for it.Related Tickets & Documents
None.