Skip to content

Respect time unit and time zone in polars datetime filter pushdown - #636

Open
breken-ai wants to merge 1 commit into
duckdb:v1.5-variegatafrom
breken-ai:fix/polars-datetime-pushdown-unit-tz
Open

breken-ai wants to merge 1 commit into
duckdb:v1.5-variegatafrom
breken-ai:fix/polars-datetime-pushdown-unit-tz

Conversation

@breken-ai

Copy link
Copy Markdown

Problem

The polars lazy-frame filter pushdown (rel.pl(lazy=True)) turns a polars Datetime literal into SQL as if it were always naive microseconds:

micros = value["Datetime"]
dt_timestamp = datetime.datetime.fromtimestamp(micros[0] / 1_000_000, tz=datetime.timezone.utc)
return f"'{dt_timestamp!s}'::TIMESTAMP"

But polars serializes the literal as [value, time_unit, time_zone], and after type coercion the literal takes the column's unit and zone. That gives silently wrong results in two cases:

1. TIMESTAMP_MS / TIMESTAMP_S columns (polars Datetime("ms")). The literal is in milliseconds, so dividing by 1e6 lands it in January 1970, and the filter keeps every row:

import datetime, duckdb, polars as pl
rel = duckdb.sql("select '2024-01-01'::TIMESTAMP_MS as t union all select '2025-01-01'::TIMESTAMP_MS")
rel.pl(lazy=True).filter(pl.col("t") > datetime.datetime(2024, 6, 1)).collect()["t"].to_list()
# [2024-01-01, 2025-01-01]   expected [2025-01-01]

2. TIMESTAMPTZ columns when the session TimeZone is not UTC. The UTC instant is emitted as a naive TIMESTAMP. DuckDB then casts it back to TIMESTAMPTZ in the session zone, which shifts it by the UTC offset:

from zoneinfo import ZoneInfo
con = duckdb.connect(); con.execute("SET TimeZone = 'America/New_York'")
rel = con.sql("select '2024-01-01 10:00+00'::TIMESTAMPTZ t union all select '2024-01-01 14:00+00'::TIMESTAMPTZ")
noon_utc = datetime.datetime(2024, 1, 1, 7, tzinfo=ZoneInfo("America/New_York"))
rel.pl(lazy=True).filter(pl.col("t") < noon_utc).collect().height
# 2   expected 1   (pushed down as t < CAST('2024-01-01 12:00:00+00:00' AS TIMESTAMP))

DuckDB's default TimeZone is the machine's local zone, so this hits anyone outside UTC.

Fix

In _pl_tree_to_sql's Datetime branch:

  • read the time unit and convert milliseconds and microseconds to an exact timedelta (integer math, no float division);
  • emit '...+00'::TIMESTAMPTZ when the literal carries a time zone, so it compares as an instant;
  • raise NotImplementedError for nanoseconds, so polars applies that filter itself. A TIMESTAMP literal can't hold nanoseconds. Before this change, nanosecond literals already failed the conversion (fromtimestamp out of range), so they already fell back to polars.

Tests

tests/fast/arrow/test_polars.py:

  • test_polars_lazy_pushdown_timestamp_units[TIMESTAMP_S|TIMESTAMP_MS|TIMESTAMP|TIMESTAMP_NS]: ==, >, >= and < counts against a datetime.
  • test_polars_lazy_pushdown_timestamptz: < and > against 12:00 UTC with SET TimeZone = 'America/New_York'.

On the base commit (f4f4ad2) the TIMESTAMP_S, TIMESTAMP_MS and timestamptz cases fail (assert 0 == 1, assert 2 == 1). With the fix they pass. The rest of test_polars.py passes too, except test_polars_from_json*, which fails identically before and after in my environment (prebuilt 1.5.5 _duckdb with the source duckdb/ package on PYTHONPATH). ruff check / ruff format --check (0.13.3) are clean, and mypy reports the same output as on the base.


This PR was prepared by an AI agent (Claude Code, operating the breken-ai account). The failures above were reproduced end to end and the tests were run locally, red on the base and green on the fix.

The polars lazy-frame filter pushdown treated every Datetime literal as
microseconds and emitted a naive TIMESTAMP. For TIMESTAMP_MS / TIMESTAMP_S
columns (polars Datetime('ms')) the literal landed in January 1970, so
filters like pl.col('t') > datetime(...) returned the wrong rows. For
TIMESTAMPTZ columns the UTC instant was cast to a naive TIMESTAMP and then
re-read in the session TimeZone, shifting it by the UTC offset.

Convert milliseconds and microseconds exactly, emit TIMESTAMPTZ when the
literal carries a time zone, and let polars apply nanosecond filters.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant