Skip to content

fix: keep decimal precision for SLT tests - #24934

Open
theirix wants to merge 9 commits into
apache:mainfrom
theirix:decimal-precision-trailing-zero
Open

fix: keep decimal precision for SLT tests#24934
theirix wants to merge 9 commits into
apache:mainfrom
theirix:decimal-precision-trailing-zero

Conversation

@theirix

@theirix theirix commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Provide full precision for decimals in SLT tests.
Separate float formatting (round to 12/15 digits) from decimal formatting (avoid rounding)

What changes are included in this PR?

  • For floats, round to 12 digits by default and 15 for Spark (historically), via a generic function

  • For decimals, avoid rounding entirely. Emit any trailing zeroes to SLT output, as they are part of decimals. Rely on format_decimal directly

  • For Postgres mode with decimal_to_str - emit BigDecimal representation of numerics directly

  • Remove big_decimal_to_str helper with rounding behaviour

  • A few amends to pg_compat tests. I introduced casts in pg_compat tests to fit pg_compat mode with explicit types, either double precision or numeric(p,s). Postgres treats literals as numeric and so prints without rounding. For some UDF (avg, exp), it was required to cast the output to float explicitly, because Postgres widens inputs and outputs to decimal. In DataFusion, it produces floats for float inputs.

  • Refines sqllogictest handles decimals inconsistently #23160 - CC @AdamGS

What is the testing strategy for this PR?

  • An SLT test case for the issue - now it produces the expected decimal with full precision
  • Reworked unit tests to test float and decimal conversion, but not an intermediate big_decimal_to_str
  • Regenerated SLTs with --complete to introduce trailing zeroes - a huge inevitable change

Are there any user-facing changes?

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) auto detected api change Auto detected API change labels Sep 3, 2026
@theirix theirix changed the title Add reproducing case fix: keep decimal precision for SLT tests Sep 3, 2026
Make an explicit type, either double precision or numeric(p,s), to fix tests, including pg_compat mode.
Postgres treats literals as numerics and so prints without rounding
@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Sep 4, 2026
@codecov-commenter

codecov-commenter commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.61%. Comparing base (09a2aff) to head (4563d1f).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24934    +/-   ##
========================================
  Coverage   81.60%   81.61%            
========================================
  Files        1123     1124     +1     
  Lines      411514   411978   +464     
  Branches   411514   411978   +464     
========================================
+ Hits       335811   336225   +414     
- Misses      55924    55941    +17     
- Partials    19779    19812    +33     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@theirix
theirix marked this pull request as ready for review September 4, 2026 21:20

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @theirix , LGTM!

} else {
big_decimal_to_str(BigDecimal::from_str(&value.to_string()).unwrap(), None)
}
float_to_str(value, 12)

@jayzhan211 jayzhan211 Sep 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering why 12 and 15 were chosen here. Here's an explanation from Claude — maybe we could add a comment capturing this so it's clear to the next reader.

/// Decimal places floats are rounded to before comparison.
///
/// Note this is decimal *places*, not significant digits: `BigDecimal::round`
/// counts from the decimal point, so how much precision survives depends on the
/// magnitude of the value. Near 1.0 this keeps ~12-13 significant digits and
/// discards the tail where float noise lives — summation order across
/// partitions, FMA, platform libm differences — which is what makes results
/// reproducible across machines and partition counts. Above ~1e5 it is
/// effectively a no-op (the shortest round-trip repr has fewer than 12
/// fractional digits); below ~1e-12 values collapse to `0`.
///
/// The value is empirical, inherited from the original Postgres-compatibility
/// runner (#4834), not derived from a spec. It is kept because floats are
/// inexact and rounding away their last digits is deliberate: for a test
/// harness, determinism matters more than display fidelity. Decimals are exact,
/// have no noise to suppress, and are deliberately *not* rounded — see
/// [`arrow_decimal_to_str`].
const FLOAT_ROUND_DIGITS: i64 = 12;

/// Spark's expected values under `test_files/spark` were recorded from real
/// Spark output, which carries more digits than [`FLOAT_ROUND_DIGITS`]
/// preserves. Raised to 15 in #15168 so they stop being truncated; this is not
/// a Spark rule, and Spark does not specify decimal places.
const SPARK_FLOAT_ROUND_DIGITS: i64 = 15;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I added constants and some explanations to the code and README.

12 was a rational choice when introducing Postgres tests to handle rounding the smallest 16-bit type in
#4834 (comment)

12 is chosen to pass the existing set of tests. I think it could produce errors, for example, when rounding f16 to 12 digits. I would probably use 3 (or 4) decimal digits if high precision is not required for Postgres compatibility tests.

And 15 relates to expm1 Spark behaviour in the mentioned ticket 15168

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decimal values in SLT tests lose display precision if they have trailing zeros

3 participants