Skip to content

Pwmj metrics accounting - #24956

Open
SubhamSinghal wants to merge 2 commits into
apache:mainfrom
SubhamSinghal:pwmj-metrics-accounting
Open

Pwmj metrics accounting#24956
SubhamSinghal wants to merge 2 commits into
apache:mainfrom
SubhamSinghal:pwmj-metrics-accounting

Conversation

@SubhamSinghal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #17427.

Rationale for this change

PiecewiseMergeJoinExec's metrics accounting had several gaps left over from earlier PRs in the epic:

  • The classic-join stream (Left/Right/Full/Inner) never routed poll_next through BaselineMetrics::record_poll, so output_rows, output_bytes, and output_batches stayed at 0 in EXPLAIN ANALYZE regardless of how many rows the join actually produced.
  • join_time was never measured on either the classic or existence-join stream — only build_time was timed, so elapsed_compute understated total operator cost for large probe sides.
  • probe_hit_rate/avg_fanout exist on the shared BuildProbeJoinMetrics struct (populated by HashJoinExec) but PWMJ never populated them, always showing N/A (0/0).

What changes are included in this PR?

  • ClassicPWMJStream::poll_next now calls self.join_metrics.baseline.record_poll(poll), matching the existence-join stream.
  • join_time is timed around the actual comparison work: resolve_classic_join and the ProcessUnmatched bitmap/take pass on the classic path; extreme_key + mark_matched_buffered_rows on the existence path.
  • probe_hit_rate/avg_fanout are populated for classic join, using the range size (buffered_len - buffer_idx) already computed at each match as the fanout.
  • probe_hit_rate is populated for existence join: a streamed batch counts as a hit if its extreme key lowers the shared watermark, a miss otherwise. avg_fanout is intentionally left unset for existence join — its watermark-based semantics don't have a natural per-row fanout equivalent.
  • Added a metrics-focused test to each stream module: classic_join::tests::inner_join_records_output_and_probe_metrics and existence_join::tests::probe_hit_rate_counts_batches_that_advance_the_watermark, both hand-derived and verified against actual runs.

Are these changes tested?

Yes — two new unit tests

Are there any user-facing changes?

EXPLAIN ANALYZE on a PiecewiseMergeJoinExec plan now reports accurate output_rows/output_bytes/output_batches/join_time/probe_hit_rate/avg_fanout instead of zeros/N/A. No API or behavior change to query results.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Sep 5, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.31776% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.61%. Comparing base (35f58f5) to head (d24e66a).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...n/src/joins/piecewise_merge_join/existence_join.rs 88.03% 4 Missing and 10 partials ⚠️
...lan/src/joins/piecewise_merge_join/classic_join.rs 88.65% 0 Missing and 11 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24956    +/-   ##
========================================
  Coverage   81.61%   81.61%            
========================================
  Files        1124     1124            
  Lines      411978   412288   +310     
  Branches   411978   412288   +310     
========================================
+ Hits       336236   336499   +263     
- Misses      55936    55955    +19     
- Partials    19806    19834    +28     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants