Repository navigation
Speed up producer batch acknowledgement tracking - #300
Open
davidthorne72 wants to merge 1 commit into
Open
davidthorne72 wants to merge 1 commit into
davidthorne72 wants to merge 1 commit into
Conversation
Index batches by ledger, entry, and partition and count remaining unacknowledged indices so ACKs no longer scan batches or bit arrays under the receive lock. Preserve partial progress on redelivery. Add regression tests and repeatable lookup, batch-size, and concurrent ACK/receive benchmarks. Refs apache#298
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partially addresses #298; complements the independent bulk-ACK correctness PR #299. Both branches are based directly on upstream master, and this PR does not depend on #299.
Description
Index outstanding producer batches by ledger, entry, and partition instead of searching a linked list under the receive/ACK lock. Track the number of unacknowledged indices alongside the bit array instead of rescanning every bit after each ACK. Remove completed batches by key, and retain partial ACK progress when an incomplete batch is redelivered.
The shared lock remains; the work performed while holding it is now constant-time on average per batched ID.
Regression
No release regression is established. New unit tests cover cross-ledger/partition identity, out-of-order and duplicate indices, incomplete and completed batch redelivery, clearing on seek, disabled tracking for readers, and concurrent completion. The incomplete-redelivery test fails on the original implementation because a duplicate tracking entry remains after the first completion.
Testing
These are synthetic tracker benchmarks using a repeated partial-batch index; the parallel case polls an empty receive queue. They demonstrate scaling and reduced time holding the shared lock, not measured production throughput or a production contention percentage.