[18808] Add schema-diff helper and column deletion ledger

[Vamsi-klu]

PR flow

Flow of column deletion ledger: create entry, add, persist, reconcile, check dirty segments via CRC.

flowchart TD
  N0["Create ColumnDeletionEntry #40;PREPARED#41; #40;F3#41;"]:::stAdded
  N1["Add entry to ColumnDeletionMetadata #40;F4#41;"]:::stAdded
  N2["Persist ledger via CAS write #40;F5#41;"]:::stAdded
  N3["Reconcile PREPARED entry after schema write #40;F4#41;"]:::stAdded
  N4["Determine if segment is dirty for deletion #40;F2#41;"]:::stAdded
  N5["Check for successful refresh CRC marker #40;F2#41;"]:::stAdded
  N6["Compute schema diff for deleted#47;added columns #40;F9#41;"]:::stAdded
  N0 -->|"entry passed"| N1
  N1 -->|"metadata passed"| N2
  N2 -->|"calls toZNRecord#47;peekFormatVersion"| N1
  N1 -->|"uses schema diff for duplicate check"| N6
  N3 -->|"calls withState"| N0
  N4 -->|"calls hasSuccessfulRefreshMarker"| N5
  classDef stAdded fill:#dafbe1,stroke:#1a7f37,color:#1f2328,stroke-width:2px
  classDef stModified fill:#fff8c5,stroke:#9a6700,color:#1f2328,stroke-width:2px
  classDef stRemoved fill:#ffebe9,stroke:#cf222e,color:#1f2328,stroke-width:2px
  classDef stUnchanged fill:#f6f8fa,stroke:#656d76,color:#1f2328,stroke-width:1px

Loading

AI-generated · Green: added · Yellow: modified · Red: removed · Gray: existing

Diff evidence

• F2: pinot-common/src/main/java/org/apache/pinot/common/metadata/columndeletion/ColumnDeletionDirtySegmentPredicate.java — after
• F3: pinot-common/src/main/java/org/apache/pinot/common/metadata/columndeletion/ColumnDeletionEntry.java — after
• F4: pinot-common/src/main/java/org/apache/pinot/common/metadata/columndeletion/ColumnDeletionMetadata.java — after
• F5: pinot-common/src/main/java/org/apache/pinot/common/metadata/columndeletion/ColumnDeletionMetadataAccessHelper.java — after
• F9: pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaDiff.java — after

• Regenerate PR flow

Ledger-only foundation for first-class column deletion (#18808). This is not a public delete API and does not replace #18831.

What this adds

• SchemaDiff in pinot-spi: dedicated helper for deleted / added / retained-incompatible columns. Not an overload of Schema.isBackwardCompatibleWith. Primary-key equality stays exact-name. ignoreCase uses Locale.ROOT for column-name matching only.
• PropertyStore path /COLUMN_DELETION_METADATA/<tableNameWithType>.
• Format-versioned ledger (PREPARED → PENDING / RECLAIMING → COMPLETE / FAILED). Newer stored formats are refused. write(..., -1) is create-only; updates CAS the version from the last read.
• Dirty-set helper: RefreshSegmentTask.time is not a success marker. Equal ctime is dirty. An empty live set is not COMPLETE. Callers may pass a physical-presence override. This PR does not read segment files.

What this does not add

• No REST or client allowColumnDeletion
• No reclaimDeletedColumnsOnReload
• No BaseDefaultColumnHandler REMOVE expansion
• No schema-update path that actually deletes a column

Tests

Unit tests in pinot-spi (SchemaDiffTest, SchemaTest) and pinot-common (ColumnDeletionMetadataTest, ColumnDeletionMetadataAccessHelperTest, ColumnDeletionDirtySegmentPredicateTest).

Compatibility

Additive only. New ZK prefix is unused until a later writer exists. Mixed-version controllers that cannot parse a newer ledger format refuse to overwrite it.

Labels: feature, release-notes (new public SchemaDiff type in pinot-spi).

[Vamsi-klu]

cc @sourabh-27 @xiangfu0 ledger-only, no public delete API

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.82063% with 90 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.73%. Comparing base (30f5d20) to head (239f614).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
...etadata/columndeletion/ColumnDeletionMetadata.java	80.35%	21 Missing and 12 partials ⚠️
...ain/java/org/apache/pinot/spi/data/SchemaDiff.java	72.81%	25 Missing and 3 partials ⚠️
...n/metadata/columndeletion/ColumnDeletionEntry.java	69.04%	6 Missing and 7 partials ⚠️
...columndeletion/ColumnDeletionPhysicalOverride.java	72.41%	5 Missing and 3 partials ⚠️
...mndeletion/ColumnDeletionMetadataAccessHelper.java	88.88%	3 Missing and 2 partials ⚠️
...tion/ColumnDeletionUnsupportedFormatException.java	66.66%	2 Missing ⚠️
...ndeletion/ColumnDeletionDirtySegmentPredicate.java	97.72%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19532      +/-   ##
============================================
- Coverage     67.78%   67.73%   -0.05%     
  Complexity     1450     1450              
============================================
  Files          3490     3498       +8     
  Lines        225009   225469     +460     
  Branches      35523    35614      +91     
============================================
+ Hits         152515   152725     +210     
- Misses        60449    60690     +241     
- Partials      12045    12054       +9     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	67.73% <79.82%> (-0.05%)	⬇️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.73% <79.82%> (-0.05%)	⬇️
unittests	67.73% <79.82%> (-0.05%)	⬇️
unittests1	57.88% <79.82%> (+0.04%)	⬆️
unittests2	39.40% <0.00%> (-0.14%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.