Repository navigation
fix(ebpf): detect the HTTP/2 connection preface before other protocols - #360
mayankpande88 wants to merge 2 commits into
Conversation
The write path detected HTTP/2 first only on ports 443 and 8443. On any
other port the protocol detectors ran in order, and is_redis_query
accepts anything that starts with an uppercase letter, so the client
connection preface ("PRI * HTTP/2.0...") was taken for a Redis command
and the connection was cached as Redis. Every write until the first
server frame was read, which flips the connection to HTTP/2, was lost.
Go and gRPC clients write their first request's headers before reading
the server's SETTINGS, so those headers were lost on every new HTTP/2
connection to a port other than 443/8443, internal gRPC included. They
carry most of the HPACK dynamic table's insertions, so the decoder was
missing them for the life of the connection: every later request that
referenced them failed to decode.
The preface is unambiguous, so it is now checked before the other
detectors on every port.
There was a problem hiding this comment.
Code Review
This pull request updates the HTTP/2 detection logic in trace_enter_write to check for the client connection preface on any port, preventing connections from being misidentified as Redis. The review feedback suggests optimizing the logical condition to avoid redundant checks and save eBPF verifier and runtime instructions on the write hot path.
| if (conn->dport != 53 && http2_detection_allowed(conn) && | ||
| (is_likely_http2_port(conn->dport) || is_http2_client_preface(payload, size)) && | ||
| looks_like_http2_frame(payload, size, METHOD_HTTP2_CLIENT_FRAMES)) { |
There was a problem hiding this comment.
Since is_http2_client_preface(payload, size) being true guarantees that looks_like_http2_frame(payload, size, METHOD_HTTP2_CLIENT_FRAMES) will also return 1 (as it immediately checks and returns 1 on the preface), we can simplify the logical expression.
By rewriting the condition to is_http2_client_preface(payload, size) || (is_likely_http2_port(conn->dport) && looks_like_http2_frame(payload, size, METHOD_HTTP2_CLIENT_FRAMES)), we avoid:
- Calling
is_http2_client_prefacetwice when the preface is present on a non-standard port. - Calling
looks_like_http2_frameentirely when the preface is matched.
This reduces redundant checks and saves eBPF verifier/runtime instructions on the write hot path.
if (conn->dport != 53 && http2_detection_allowed(conn) &&
(is_http2_client_preface(payload, size) ||
(is_likely_http2_port(conn->dport) && looks_like_http2_frame(payload, size, METHOD_HTTP2_CLIENT_FRAMES)))) {There was a problem hiding this comment.
Applied in a6631d9: the preface check now short-circuits, and frame detection runs only on the 443/8443 path.
The write probe moved past the stack check, so ebpf.go is regenerated. The preface check also short-circuits the frame detection it implies.
|
Folded into #364 with the rest of this stack, so it can be reviewed and merged as one unit (squash-only merges made each stacked merge conflict). The commit is unchanged there; its description and review thread here stay as the detailed write-up. |
Summary
The write path checked for HTTP/2 first only on ports 443 and 8443. On any other port the protocol detectors ran in order.
is_redis_queryaccepts anything starting with an uppercase letter, so the client connection preface (PRI * HTTP/2.0…) was taken for a Redis command and the connection was cached as Redis.Every write until the first server frame was read (which flips the connection to HTTP/2) was lost. Go and gRPC clients write their first request's headers before reading the server's SETTINGS. So on every new HTTP/2 connection to a port other than 443/8443, internal gRPC included, the first request's headers were lost.
Those headers carry most of the HPACK dynamic table's insertions:
:authority,user-agent,content-type, the first:path. The decoder was missing them for the life of the connection, and every later request that referenced them failed to decode.The preface is 24 fixed bytes and can't be mistaken for anything else, so it is now checked before the other detectors on every port.
Stacked on #359.
Engineering detail
TestProgramsLoadandTestGoTLSFdWalkpass on 5.10 and 6.1.sys_enter_sendmsgprocesses 178,069 verifier instructions.Local e2e: built agent images with and without this change, each with a local-only patch that dumps the HTTP/2 events the parser receives, and ran them in a local Docker VM (kernel 6.10) against a Go HTTP/2 client and server on port 9443. Without the change, the first HTTP/2 client event was a mid-stream write: the preface, the SETTINGS ACK and the first request chunk never reached the agent. With it, the first event is the preface and the whole client stream replays as aligned frames. In the same setup, with a Go net/http client making ~1.6M requests per 60s run: the parent image gave 1.5M–3M HPACK decode errors. This change together with #357 (decoder) and the stream-cap PR (#361) gave 0 errors in both runs, and every request the client made was decoded. This change alone does not remove the errors, because of the table-size-update cascade #357 fixes.
go vetand the eBPF load tests pass.