feat(storage): extend read hedging to mid-stream chunk reads

[ajayky-os]

Motivation

#16344 added speculative hedging to ReadObject(), but only for the stream open. Once an attempt won the race, every subsequent Read() was a plain passthrough to that child for the life of the download.

That covers TTFB, which is where most tail latency lives, but it leaves a real gap: a connection that is healthy at open can stall later. For a large object read in chunks, a single stalled connection mid-download costs as much as a slow open, and the client had no way to recover short of the retry policy firing on an actual error — a stall that never errors just blocks.

This PR extends hedging to any read in the stream.

What changed

HedgedObjectReadSource becomes stateful. It tracks the stream's position the same way RetryObjectReadSource already does when it resumes after a failure — current offset, offset direction, pinned generation, object size, and whether decompressive transcoding is in play — so it can open a new child at the stream's current position rather than at the original request offset.

Each read is timed. A read that takes longer than ReadHedgeDelayOption marks the stream as stalled, and the next read is raced: the existing active child becomes the primary attempt, and hedges are opened at the current offset, pinned to the generation observed so far. A hedge that wins replaces the active child and serves the rest of the stream. Once a read completes within the delay, the stream returns to direct reads on the caller's thread with no thread hops or copies.

To support this, ChildFactory gained the position it should open at:

using ChildFactory = std::function<StatusOr<std::unique_ptr<ObjectReadSource>>(
    std::int64_t current_offset, std::optional<std::int64_t> generation)>;

StorageConnectionImpl::ReadObject() implements it by applying the same request rewrite RetryObjectReadSource::Read() applies on resume (ReadLast / ReadFromOffset, plus Generation when known), so hedged and retried reads resume through identical logic.

Where racing is deliberately skipped

Mid-stream racing is not always safe or useful, so it is bypassed:

Case	Reason
Decompressive transcoding (gunzipped)	Byte ranges are not honored; a hedge would restart from the first byte
Read larger than MaximumHedgeBufferOption	Each attempt stages its own copy, so a large read would multiply memory use
Stream is at the end of the requested data	A hedge would ask for an empty or inverted range
Stream is not stalled	A healthy stream keeps the zero-cost direct path

The stream open is always raced, as before.

Error handling changes

Important

These are behavioral changes beyond mid-stream hedging and deserve their own look.

RaceState previously resolved only on the primary's open error; a hedge that failed was silently ignored, and if every attempt failed the caller could be left waiting. It now:

• Tracks outstanding attempts. The last attempt to retire resolves the race with the collected error, so a race where every attempt fails always terminates.
• Prefers the primary's error. The primary describes the stream the caller is actually reading, so its error takes precedence over whatever a hedge happened to fail with last.
• Fails fast on a permanent primary error. A hedge cannot fix kNotFound or kPermissionDenied, so the caller is not held until every in-flight hedge has exhausted its own retry budget.

Compatibility

• No new public options. Reuses EnableReadHedgingOption, ReadHedgeDelayOption, MaxReadHedgesOption, and MaximumHedgeBufferOption.
• No change when hedging is disabled. The non-hedged path calls the same child factory once, at the request's starting position — identical to the previous retry_source_factory().
• Internal-only: HedgedObjectReadSource and ChildFactory live in storage::internal.
• The staging buffer of a winning attempt is retained and reused by the next race, so repeated racing does not re-allocate per read.

Testing

google/cloud/storage:internal_hedged_object_read_source_test grows from 14 to 26 cases, covering generation pinning, gunzip bypass, ReadLast offset tracking and clamping, the three end-of-stream variants, stall detection and the return to direct reads, hedge-pool exhaustion, and the all-attempts-fail paths.

//google/cloud/storage:internal_hedged_object_read_source_test   PASSED
//google/cloud/storage:internal_connection_impl_test             PASSED
bazel build //...                                                4,130 targets, OK

Areas worth reviewer attention

1. Position bookkeeping parity with RetryObjectReadSource. UpdateState() duplicates the offset/generation/gunzip logic in RetryObjectReadSource::HandleResult(). Is the duplication acceptable, or should the two share a small helper? They must not drift.
2. ReadLast clamping. ReadLast(N) with N larger than the object returns the whole object, so the remaining count has to be bounded by the object size before a hedge is opened with it — otherwise the hedge covers the whole object again. Covered by ReadLastLargerThanObjectClampsOffset.
3. Re-race policy. A read is re-raced whenever the previous one exceeded the hedge delay. Whether that is the right trigger for streams that are uniformly slow rather than stalled is worth a second opinion.

[gemini-code-assist]

Code Review

This pull request extends the HedgedObjectReadSource to support hedging of mid-stream reads when a stall is detected, rather than only hedging the initial stream open. It introduces tracking of the current offset, direction, and generation to allow new hedge attempts to resume reading from the correct position. The feedback focuses on style guide compliance regarding type deduction, specifically requesting the replacement of auto with explicit types where it obscures StatusOr return types or primitive values.

[codecov]

Codecov Report

❌ Patch coverage is 98.57513% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.34%. Comparing base (11acb31) to head (0c38ad2).

Files with missing lines	Patch %	Lines
...loud/storage/internal/hedged_object_read_source.cc	96.77%	4 Missing ⚠️
google/cloud/storage/internal/connection_impl.cc	88.88%	3 Missing ⚠️
.../storage/internal/retry_object_read_source_test.cc	94.82%	3 Missing ⚠️
...storage/internal/hedged_object_read_source_test.cc	99.80%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16446      +/-   ##
==========================================
+ Coverage   92.32%   92.34%   +0.02%     
==========================================
  Files        2246     2246              
  Lines      213921   214568     +647     
==========================================
+ Hits       197504   198149     +645     
- Misses      16417    16419       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ajayky-os]

/gcbrun

[ajayky-os]

/gemini review

[gemini-code-assist]

Code Review

This pull request enhances HedgedObjectReadSource to support hedging subsequent reads mid-stream when a read stalls, rather than only hedging the initial stream open. It introduces tracking for the current offset, direction, end offset, and generation, and updates the child factory to allow opening new streams at specific positions. Additionally, it adds comprehensive unit tests to verify the mid-stream hedging behavior, stalling detection, and offset tracking. A review comment correctly identifies a style guide violation where auto is used to deduce a primitive type (std::int64_t) in hedged_object_read_source.cc.

[scotthart]

/gcbrun