Skip to content

Do not push down inf/nan float literals in the polars lazy filter - #637

Open
breken-ai wants to merge 1 commit into
duckdb:v1.5-variegatafrom
breken-ai:fix/polars-nonfinite-float-pushdown
Open

breken-ai wants to merge 1 commit into
duckdb:v1.5-variegatafrom
breken-ai:fix/polars-nonfinite-float-pushdown

Conversation

@breken-ai

Copy link
Copy Markdown

Problem

JSON has no inf/nan, so polars serializes a Float32/Float64 literal of inf, -inf or nan as null:

{"BinaryExpr":{"left":{"Column":"a"},"op":"Lt","right":{"Literal":{"Scalar":{"Float64":null}}}}}

The polars lazy-frame pushdown then returns str(value[dtype]), which is the SQL text None. DuckDB binds that as a column name, so the whole collect() fails:

import duckdb, polars as pl
lf = duckdb.sql("select * from (values (1.0::DOUBLE), (2.5)) t(a)").pl(lazy=True)
lf.filter(pl.col("a") < float("inf")).collect()
# polars.exceptions.ComputeError: caught exception during execution of a Python source,
# exception: BinderException: Binder Error: Referenced column "None" not found in FROM clause!

The same happens for > float("-inf"), == np.inf, comparisons with nan, and so on. These are common bounds in numeric filters. The parse step (SQLExpression) accepts None as an identifier, so the existing "fall back to polars" path never triggers.

A real null literal serializes differently ({"Scalar": {"Null": "Float64"}}), so a null float value always means a non-finite number. We can't tell which one it is, so the literal can't be translated.

Fix

In the scalar numeric branch of _pl_tree_to_sql, raise NotImplementedError when the value is null. The predicate is then not pushed down, and polars applies it to the batches (the existing fallback).

Tests

tests/fast/arrow/test_polars.py::test_polars_lazy_pushdown_non_finite_float[FLOAT|DOUBLE] builds a column containing 1, inf, -inf, nan and NULL. For < inf, > -inf, == inf and >= nan, it checks that the predicate is not pushed down and that the lazy result equals the eager polars result.

On the base commit (f4f4ad2) both cases fail with the BinderException above. With the fix they pass. The rest of test_polars.py passes too, except test_polars_from_json*, which fails identically before and after in my environment (prebuilt 1.5.5 _duckdb with the source duckdb/ package on PYTHONPATH). ruff check / ruff format --check (0.13.3) are clean, and mypy reports the same output as on the base.


This PR was prepared by an AI agent (Claude Code, operating the breken-ai account). The failure above was reproduced end to end and the tests were run locally, red on the base and green on the fix.

JSON has no inf/nan, so polars serializes a Float32/Float64 literal of
inf, -inf or nan as null. The pushdown turned that into the SQL text
None, which DuckDB binds as a column name, so a lazy filter such as
pl.col('x') < float('inf') failed the whole collect() with
'Referenced column "None" not found'.

Treat a null float value as not pushable so polars applies the filter.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant