[core][spark] Deduplicate a replayed Structured Streaming micro-batch

[zhuxiangyi]

Purpose

Closes #9666.

Structured Streaming delivers exactly-once only if the sink is idempotent for a repeated batch id.
When a query fails after the sink returns from addBatch but before Spark records the batch as
completed, the restarted query replays that micro-batch with its original batch id.

PaimonSink received the batch id but used it only to pace full compaction, and committed through
table.newBatchWriteBuilder(), whose commit user is a fresh random UUID per builder and whose
commit identifier is always BatchWriteBuilder.COMMIT_IDENTIFIER = Long.MAX_VALUE. Neither of the
dimensions Paimon deduplicates on could identify a replay, so the batch was committed a second
time: every row duplicated in an append-only table, and silently wrong values in an aggregation
merge-engine table (writing (1, 10) and replaying that batch yields v = 20).

The machinery already exists in core and is what the Flink sink uses; the Spark sink simply took
the batch write path.

Tests

PaimonSinkIdempotencyTest (new, 7 cases), each asserting the correct behaviour so that it fails
without the fix:

• a replayed micro-batch of an append-only table, driven end to end by deleting the commit log
  entry of the batch, which is exactly the checkpoint state a driver failure leaves behind;
• the same when only the query id is available, i.e. the checkpoint location never reaches the sink
  options because it comes from spark.sql.streaming.checkpointLocation;
• a replay of a batch that is not the first one of the query;
• a replay in complete output mode;
• write.stream.commit-user as an option of the writer and as a spark.paimon. session conf;
• addBatch called twice with the same batch id through the API directly.

Two of them assert the prefix of the commit user recorded in the snapshot, so that the case which
is meant to exercise the query id derivation cannot pass through the checkpoint derivation.

The full set of Spark streaming suites was run on the spark3 and spark4 profiles (Spark 3.2, 3.4,
3.5, 4.1): 34 suites, 349 tests.

API and Format

BatchWriteBuilderImpl gains withCommitUser, and its newCommit() return type is narrowed from
BatchTableCommit to InnerTableCommit. The narrowing keeps the caller in the connector free of a
downcast that could only fail at runtime; it is source compatible, and BatchWriteBuilderImpl has
no subclasses.

InnerTableCommit gains checkFilesExistence(boolean). filterAndCommit verified that every file
it is about to commit still exists, which guards a committable restored from an engine's state that
may reference files deleted long ago. A caller filtering a committable it has just produced knows
those files exist, so the Spark sink turns the check off; otherwise every micro-batch would pay a
file listing proportional to the number of files it wrote. The default is unchanged, so Flink keeps
the check.

Documentation

docs/docs/spark/structured-streaming.md gains an "Exactly-once" section covering how the commit
user is derived, the new write.stream.commit-user option, and the limits: starting from a new
checkpoint location gives a query a new commit user, the data files of a skipped replay are left to
orphan file cleaning, and a postpone bucket table committing through the staged committer cannot
deduplicate and logs a warning per micro-batch. The generated option reference is regenerated.

[JingsongLi]

Two issues found in the checkpoint identity and commit maintenance lifecycle. The existing seven Spark 3 tests pass; additional checkpoint edge-case tests and a focused maintenance probe reproduce the issues below.

[JingsongLi]

[P1] Prefer the persisted query ID over the checkpoint path

If a checkpoint is deleted and a new query starts at the same path, Spark assigns a new query ID and restarts batch IDs at 0. This code nevertheless reuses the previous commit user, so filterCommitted silently skips fresh batches whose IDs are at or below the previous query's last committed ID. I reproduced this with different input in the new query: the query completed successfully, but the table contained only the old row instead of both rows.

The reverse also fails: restarting the same checkpoint with only a trailing / added to its path changes the commit user despite an unchanged query ID. Replaying batch 0 then produced two rows instead of one.

Please keep the explicit override, but prefer the persisted Spark query ID for the default identity and use the checkpoint path only as a fallback. Add coverage for both a fresh query reusing a checkpoint path and an existing query using an equivalent path spelling.

[zhuxiangyi]

Confirmed both, and reproduced both before changing anything.

A fresh query at a reused location: the first query wrote [1,old] under spark-checkpoint-b492463a…; after deleting the checkpoint, a new query writing (2,"new") derived the identical user, and the table still held only [1,old] — the new row was skipped as an already committed replay, with the query reporting success. It is not limited to the first batch: the comparison is against the previous run’\s last committed identifier, so a new query silently loses every batch up to that id.

The trailing separator: spark-checkpoint-ae4544b6… became spark-checkpoint-9ff6c207… for the same checkpoint, and the replayed batch produced two rows.

The underlying mistake was binding the identity to where the checkpoint is stored rather than to which incarnation of it is running, so it could come apart in both directions. The failure modes are asymmetric — a fresh identity for the same query duplicates, a stale identity for a new query loses data — which is the stronger argument for your ordering, so it is now: explicit override, then the persisted query id, then the location, which is only reachable outside a stream execution (a direct addBatch call has no query id).

Both cases you named are covered: "a new query reusing a checkpoint location must not skip its batches" and "an equivalent spelling of the checkpoint location keeps the identity", asserting the commit user differs in the first and is unchanged in the second.

Since the query id is now the default identity rather than a fallback, I also ran the suite on the 3.2, 3.4, 3.5, 4.0 and 4.1 modules to confirm the property is populated on every supported version. That surfaced an unrelated problem of my own: the suite left a second table behind, which Spark 3.2 cannot drop because its dropNamespace has no cascade overload. Fixed with withTable.

[JingsongLi]

[P2] Preserve the batch maintenance lifecycle when filtering commits

Unlike commit(List), filterAndCommit does not set TableCommitImpl.batchCommitted. Maintenance therefore runs through the streaming executor wrapper, but this writer still closes and discards the committer immediately after each batch. With snapshot.expire.execution-mode=async, close() calls shutdownNow() and can interrupt snapshot expiration before it finishes. Even with the default synchronous mode, the wrapper catches maintenance exceptions and stores them for the next commit; because this instance is discarded, those failures are never propagated to the caller.

A focused probe against the built classes reproduced both the asynchronous interruption and the loss of synchronous error propagation. The existing testBatchWriteAsyncExpireFallbackToSync also establishes that a batch committer must finish maintenance before closing.

Please preserve the one-shot batch maintenance semantics in the filtered commit path, or explicitly wait for maintenance and propagate its failure before closing the committer.

[zhuxiangyi]

Confirmed. maintain branches on batchCommitted, which only checkCommitted() sets, so the filtered path left maintenance to the executor while the writer kept the one-shot lifecycle of a batch committer — the two halves no longer matched.

The asynchronous half reproduced clearly. With snapshot.expire.execution-mode=async and min/max retained set to 1, four micro-batches left snapshots=4 earliest=1 latest=4: shutdownNow() drained the queued task, so expiration did not run at all rather than being cut short partway. For a long-running streaming write that means snapshots grow without bound.

The synchronous half I verified by reading rather than by test: the wrapper stores the failure in maintainError, which is only rethrown at the start of the next maintain, and this committer is discarded before there is one. Rather than test the swallowing, I made it structurally impossible.

InnerTableCommit now has inlineMaintenance(boolean): maintenance runs on the committing thread and its failure is thrown to the caller, which is what commit(List) already gets from batchCommitted. The Spark sink asks for it on the filtered path. The default is unchanged, so Flink keeps the executor. testBatchWriteAsyncExpireFallbackToSync was the right reference — the invariant it fixes is exactly the one I had broken, and the new test asserts the same property through the streaming sink.

Worth noting for the record: expiration now runs inline on each micro-batch, which restores the behaviour before this PR rather than adding cost. Making it asynchronous again would mean keeping one committer alive across batches, the way CommitterOperator does; that is a larger change than this fix and I have left it out.

[JingsongLi]

The original two findings are addressed in the inspected code: the persisted query ID is now preferred, and inlineMaintenance restores the one-shot commit lifecycle. Replayed micro-batch deduplication has clear value. One direct-postpone path still bypasses the promised replay behavior; details are inline.

A focused local-filesystem core probe used the actual builder/commit sequences: normal stable-user overwrite replay stayed at snapshot 1; direct postpone replay created snapshot 2, both with its default identity and when only the user was stabilized. I did not rerun the full Spark streaming suite.

[JingsongLi]

[P2] Define replay handling for the direct postpone committer too

With bucket=-2, postpone.default-bucket-num=1, and complete output mode, each overwrite takes this direct branch and never reaches the staged-path limitation warning. The actual PostponeFixedBucketWriteBuilder has its own random commit user, bypassing the stable user configured on writeBuilder. A replay therefore commits another overwrite snapshot instead of being a no-op. I reproduced snapshot 1 becoming snapshot 2; the ordinary stable-user overwrite control stayed at snapshot 1. This does not imply duplicate visible rows in complete mode, but it repeats a commit and its side effects.

Propagating the user alone is insufficient: this builder also resets commit.strict-mode.last-safe-snapshot to the current latest snapshot on each attempt, and filterCommitted searches only after that bound. The second probe with a stable user still committed snapshot 2. Handle replay lookup independently of that conflict bound and pass the stable identity to the actual committer, or explicitly exclude and warn for the direct path alongside the staged path. Please cover complete-mode replay with this configuration.

[zhuxiangyi]

Confirmed and reproduced first: with bucket = -2, postpone.default-bucket-num = 1 and complete mode, replaying batch 0 took the table from snapshot 1 to snapshot 2, exactly as you saw. Both halves of your analysis held up on inspection.

Fixed in 587ad9c along the first line you suggested:

• PostponeFixedBucketWriteBuilder gained withCommitUser, mirroring the one on BatchWriteBuilderImpl, and the direct builder is created with the stable user.
• The direct path looks the replay up itself, with latestSnapshotOfUser unbounded, and commits under the batch id only when nothing at or above it has been committed. The strict mode bound is left alone: it is what conflict detection needs, and I traced its use in filterCommitted to [core] Optimize FileStoreCommitImpl#filterCommitted when commit strict mode is set #7239/[core] Fix performance issue in FileStoreCommitImpl#filterCommitted #7275, a lookup optimisation whose premise is a commit user that never outlives its base snapshot. That premise is exactly what a stable user breaks, so the lookup for this path has to be independent of it rather than the bound relaxed for everyone.

The test for this configuration also commits a later batch after the replay and checks it lands, so that a higher batch id cannot be mistaken for a replay — the failure mode in that direction is silent data loss, which is worse than the duplicate.

Two things worth recording. First, while looking for a way to observe maintenance on this path I found that PrimaryKeyFileStoreTable.newExpireRunnable() returns null for postpone bucket tables, so snapshot expiration is never part of a commit there; a probe that expects the direct path to expire snapshots will not see it, regardless of inlineMaintenance. I restructured the sink so inlineMaintenance(true) is set once above both streaming branches, where the existing expiration test on an append table guards it, and added a core test in SimpleTableTestBase that commits through filterAndCommit with a committer closed after each batch and asserts the previous snapshot is expired before the close. Second, the staged committer is still outside the replay handling and keeps its warning; I did not extend the change there.

The suite is at 11 cases and was run on the 3.2, 3.4, 3.5, 4.0 and 4.1 modules again.

[JingsongLi]

The stable query identity, inline maintenance and unbounded direct-postpone lookup now address the previous replay findings. The new direct path correctly avoids a second snapshot for a replay and still permits a higher batch identifier; its strict conflict guard against another writer remains intact. However, one end-to-end recovery gap remains: successful deduplication must also finish callbacks that failed after snapshot publication. The attached P2 is reproduced using the changed core classes and the built-in partition-registration callback, following the new Spark branch; it is not a full Spark/metastore integration run.

[JingsongLi]

[P2] Retry commit callbacks before acknowledging an already-published batch

For the direct postpone path (bucket=-2, postpone.default-bucket-num=1) on a partitioned table with metastore.partitioned-table=true, a commit can publish its snapshot and then fail in the built-in AddPartitionCommitCallback, for example when the metastore partition-registration RPC fails. On replay, this branch only logs and returns successfully, bypassing FileStoreCommitImpl.filterCommitted, which deliberately calls callback.retry for already-committed batches. I reproduced this with a real local Paimon table and a fail-once PartitionModification: the first call throws after snapshot 1 is published; the replay succeeds with snapshot 1 still current, registrationAttempts=1 and registered=false. The normal filterAndCommit control retries registration (attempts=2, registered=true) without another snapshot. Please preserve the unbounded deduplication lookup while also completing the pending callbacks before reporting replay success; otherwise the checkpoint can advance with its committed partition still absent from the metastore.