fix: keep decimal precision for SLT tests - #24934
Conversation
Make an explicit type, either double precision or numeric(p,s), to fix tests, including pg_compat mode. Postgres treats literals as numerics and so prints without rounding
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24934 +/- ##
========================================
Coverage 81.60% 81.61%
========================================
Files 1123 1124 +1
Lines 411514 411978 +464
Branches 411514 411978 +464
========================================
+ Hits 335811 336225 +414
- Misses 55924 55941 +17
- Partials 19779 19812 +33 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jayzhan211
left a comment
There was a problem hiding this comment.
Thanks @theirix , LGTM!
| } else { | ||
| big_decimal_to_str(BigDecimal::from_str(&value.to_string()).unwrap(), None) | ||
| } | ||
| float_to_str(value, 12) |
There was a problem hiding this comment.
I was wondering why 12 and 15 were chosen here. Here's an explanation from Claude — maybe we could add a comment capturing this so it's clear to the next reader.
/// Decimal places floats are rounded to before comparison.
///
/// Note this is decimal *places*, not significant digits: `BigDecimal::round`
/// counts from the decimal point, so how much precision survives depends on the
/// magnitude of the value. Near 1.0 this keeps ~12-13 significant digits and
/// discards the tail where float noise lives — summation order across
/// partitions, FMA, platform libm differences — which is what makes results
/// reproducible across machines and partition counts. Above ~1e5 it is
/// effectively a no-op (the shortest round-trip repr has fewer than 12
/// fractional digits); below ~1e-12 values collapse to `0`.
///
/// The value is empirical, inherited from the original Postgres-compatibility
/// runner (#4834), not derived from a spec. It is kept because floats are
/// inexact and rounding away their last digits is deliberate: for a test
/// harness, determinism matters more than display fidelity. Decimals are exact,
/// have no noise to suppress, and are deliberately *not* rounded — see
/// [`arrow_decimal_to_str`].
const FLOAT_ROUND_DIGITS: i64 = 12;
/// Spark's expected values under `test_files/spark` were recorded from real
/// Spark output, which carries more digits than [`FLOAT_ROUND_DIGITS`]
/// preserves. Raised to 15 in #15168 so they stop being truncated; this is not
/// a Spark rule, and Spark does not specify decimal places.
const SPARK_FLOAT_ROUND_DIGITS: i64 = 15;There was a problem hiding this comment.
Thank you! I added constants and some explanations to the code and README.
12 was a rational choice when introducing Postgres tests to handle rounding the smallest 16-bit type in
#4834 (comment)
12 is chosen to pass the existing set of tests. I think it could produce errors, for example, when rounding f16 to 12 digits. I would probably use 3 (or 4) decimal digits if high precision is not required for Postgres compatibility tests.
And 15 relates to expm1 Spark behaviour in the mentioned ticket 15168
Which issue does this PR close?
Rationale for this change
Provide full precision for decimals in SLT tests.
Separate float formatting (round to 12/15 digits) from decimal formatting (avoid rounding)
What changes are included in this PR?
For floats, round to 12 digits by default and 15 for Spark (historically), via a generic function
For decimals, avoid rounding entirely. Emit any trailing zeroes to SLT output, as they are part of decimals. Rely on
format_decimaldirectlyFor Postgres mode with
decimal_to_str- emit BigDecimal representation of numerics directlyRemove
big_decimal_to_strhelper with rounding behaviourA few amends to pg_compat tests. I introduced casts in pg_compat tests to fit pg_compat mode with explicit types, either double precision or numeric(p,s). Postgres treats literals as numeric and so prints without rounding. For some UDF (avg, exp), it was required to cast the output to float explicitly, because Postgres widens inputs and outputs to decimal. In DataFusion, it produces floats for float inputs.
Refines sqllogictest handles decimals inconsistently #23160 - CC @AdamGS
What is the testing strategy for this PR?
big_decimal_to_str--completeto introduce trailing zeroes - a huge inevitable changeAre there any user-facing changes?