Skip to content

[spark] Convert EqualNullSafe(NaN) to isNaN in V2 filters - #9720

Merged
JingsongLi merged 1 commit into
apache:masterfrom
cxzl25:spark-v2-filter-null-safe-nan
Sep 11, 2026
Merged

[spark] Convert EqualNullSafe(NaN) to isNaN in V2 filters#9720
JingsongLi merged 1 commit into
apache:masterfrom
cxzl25:spark-v2-filter-null-safe-nan

Conversation

@cxzl25

@cxzl25 cxzl25 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Purpose

Tests

Added V2Filter: EqualNullSafe NaN to SparkV2FilterConverterTestBase.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has end-to-end value: Spark's f <=> NaN must retain NaN rows through Paimon pruning. Using IsNaN matches the existing = conversion; its row evaluation rejects NULL and its statistics evaluation remains conservative. I also checked the NOT path: IsNaN.negate() is unsupported, so conversion falls back instead of introducing incorrect negated pushdown.

No blocking issue found in this focused source review. The added FLOAT/DOUBLE SQL cases cover the intended result and nonempty scan. I have not rerun the Spark suite locally.

@JingsongLi
JingsongLi merged commit 036b889 into apache:master Sep 11, 2026
9 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants