fix(l7): make room for new HTTP/2 streams by dropping the oldest waiting one

[mayankpande88]

Summary

The HTTP/2 parser tracks at most 100 requests per connection that are waiting for their response. A request whose response it never sees waits until the stream GC, two minutes or more. That happens when the read carrying the response was cut short by truncation, or the event was lost.

Once 100 of those piled up, new streams were refused, silently. Every request on the connection was dropped until the GC ran, and no counter showed it.

At the limit the request that has waited longest is now dropped instead, and counted as a new stream_evicted stage of node_agent_http2_stage_total.

Stacked on #357.

Engineering detail

• TestHttp2ParserEvictsOldestStreamAtCap creates 120 orphaned streams, then 50 live ones with responses. All 50 complete, the oldest orphans go first, and 21 evictions are counted.
• Choosing the oldest scans the at most 100 active requests on the rare call that hits the limit.

Local e2e: in a local Docker VM (kernel 6.10), a Go net/http client made ~1.6M HTTP/2 requests per 60s run. An image with this change plus #357, #359 and #360 decoded every request the client made in both runs (1,622,425 and 1,625,464), with 0 HPACK errors. A local run with the preface fix but without this change recorded only 794k of 1.19M requests. This change was not measured alone. Unit tests and golangci-lint pass.

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to evict the oldest waiting HTTP/2 request when the maximum concurrent stream limit is reached, preventing connection stalls due to orphan streams. It adds the stream_evicted metric stage, implements the evictOldestRequest method, and includes a unit test to verify the eviction behavior. The review feedback highlights that the current O(n) traversal in evictOldestRequest could cause performance issues on the critical path, recommending an optimization to O(log n) or better (such as using a min-heap or doubly-linked list) to reduce lock contention.

[gemini-code-assist]

The cache eviction logic in evictOldestRequest currently iterates over all active requests in p.activeRequests to find the oldest one, which is an O(n) operation. Since this runs on a critical path (L7 event-processing/eBPF event handling), holding write locks during this traversal can stall event handling. Please optimize this eviction logic to O(log n) or better (for example, by maintaining a min-heap or a doubly-linked list of requests ordered by time) to minimize the time write locks are held.

References

1. Ensure cache eviction logic running on critical paths (such as L7 event-processing or eBPF event handling) is optimized (e.g., O(n log n) or better) to minimize the time write locks are held and prevent stalling event handling.

[mayankpande88]

Keeping the scan. It runs only when a connection already holds maxActiveRequests (100) requests waiting for responses, it is bounded by that 100, and it costs about a microsecond. A heap or ordered index would add bookkeeping to every stream create and complete, the hot path, to speed up a path that only runs at the limit. HTTP/2 stream IDs and kernel times also both increase, so the oldest entry is easy to find if this ever shows up in a profile.

[mayankpande88]

Folded into #364 with the rest of this stack, so it can be reviewed and merged as one unit (squash-only merges made each stacked merge conflict). The commit is unchanged there; its description and review thread here stay as the detailed write-up.