Skip to content

fix: reject NULL partition keys during partitioned writes - #24951

Draft
bvolpato wants to merge 1 commit into
apache:mainfrom
bvolpato:bvolpato/reject-null-partition-writes
Draft

fix: reject NULL partition keys during partitioned writes#24951
bvolpato wants to merge 1 commit into
apache:mainfrom
bvolpato:bvolpato/reject-null-partition-writes

Conversation

@bvolpato

@bvolpato bvolpato commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Related to #18083. Full support for NULL partition values remains covered by that issue.

Rationale for this change

Partitioned writes read partition values without checking their validity. A NULL integer partition key can be written under p=0, so reading the output turns NULL into zero and silently changes the data.

For example:

CREATE TABLE src(p INT, v INT) AS VALUES (NULL, 1), (0, 2);
COPY src TO '/tmp/null_partition/' STORED AS PARQUET PARTITIONED BY (p);

Both rows can end up in the zero partition. This PR rejects the write with NULL values are not supported for partition column 'p'.

What changes are included in this PR?

  • Check logical nulls before extracting partition keys, including NULL dictionary values referenced by valid dictionary indices.
  • Continue accepting nullable columns when the selected rows contain no NULL values.
  • Add focused unit tests and a copy.slt regression covering rejection and a filtered successful write.

What is the testing strategy for this PR?

  • Both new rejection tests fail against upstream production code at 35f58f53cde2f634c21f6370e385cf1d7e9bc55c and pass with the fix.
  • cargo test --locked -p datafusion-datasource --lib: 180 passed.
  • cargo fmt --all -- --check and git diff --check passed.
  • Draft pending all-target/all-feature Clippy, the extended workspace suite, and execution of copy.slt.

Are there any user-facing changes?

Partitioned writes containing NULL partition keys now return an execution error. Callers can filter or replace these values before writing. Validation happens per batch, so files written by earlier batches can remain after an error.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate labels Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant