[client] Resolve dynamic partition bucket counts before sending - #4309
Open
wuchong wants to merge 3 commits into
Open
[client] Resolve dynamic partition bucket counts before sending#4309wuchong wants to merge 3 commits into
wuchong wants to merge 3 commits into
Conversation
Track temporary and resolved bucket counts in each write context, and resolve the partition ID and count together before draining buffered batches. Retain batches only when their assignment remains valid; otherwise fail them with BucketRescaleException so callers can retry using the resolved layout with the same writer. Pass the current bucket count to bucket assigners and synchronize only while resolving tentative routing or switching historical targets. Preserve concurrent writes across resolved buckets, keep memory allocation outside routing locks, and let the server validate resolved batch counts. Stop the writer on server-reported invalid routing or partition-creation failures. Cover bucket-count growth and shrinkage, safe no-key batch retention, same-writer retries, concurrent routing resolution, and writer cleanup. Refs apache#4287 Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6-astra AI-Contributed/Feature: 41/41 AI-Contributed/UT: 33/33
AI-Contributed/Feature: 0/0 AI-Contributed/UT: 0/3
Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-6-astra AI-Contributed/Feature: 10/10 AI-Contributed/UT: 102/179
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Refs #4287.
A writer may buffer records before a dynamically created partition's bucket count is known. Resolve the partition ID and bucket count together before sending, so a stale table default cannot silently route records to the wrong buckets.
This PR covers tentative routing resolution and server-reported routing failures. The existing synchronous partition-existence check and shared asynchronous metadata refresh work remain follow-ups.
Brief change log
RecordAccumulatorown each physical partition's assigner, temporary bucket count, and resolved layout. Pass the current count toassignBucketandonNewBatch.BucketRescaleException; callers can resubmit the failed records with the same writer.invalidBucketRoutingTablesresult from the send path. Treat server-reportedINVALID_BUCKET_ROUTINGand partition-creation failures as fatal writer errors, including queued/in-flight batch cleanup and propagation of the original failure to later writes.Tests
BucketRoutingTest,RecordAccumulatorTest,SenderTest,HashBucketAssignerTest,StickyStaticBucketAssignerTest,PartitionBucketCountRescaleITCase,PartitionedTableITCase, andAutoPartitionedTableITCase, including sending the original resolved count after metadata changes and handling server rejection for both count growth and shrinkage../mvnw -B spotless:check validate../mvnw -B verifystopped influss-server:ReplicaTest.testBucketPhysicalStorageLocalLogSizeIncludesFollowerandtestPhysicalStorageLocalLogSizeIsScopedPerBucketboth reported expected450but actual7220. No server files are changed by this PR. An isolated rerun of all 16ReplicaTestcases passed; the full-suite failure remains unresolved, and downstream modules were not verified by that run.API and Format
Adds the public evolving
BucketRescaleExceptionfor buffered records rejected before sending. Its recovery contract permits resubmission with the same writer. Bucket assigner signature changes are internal; RPC and storage formats are unchanged.Documentation
The retry contract is documented in
BucketRescaleExceptionJavadoc. No standalone documentation changes.Generative AI assistance: OpenAI Codex (GPT-6 Astra).