Skip to content

fix(ebpf): detect the HTTP/2 connection preface before other protocols - #360

Closed
mayankpande88 wants to merge 2 commits into
fix/go-tls-write-oncefrom
fix/h2-preface-before-redis
Closed

mayankpande88 wants to merge 2 commits into
fix/go-tls-write-oncefrom
fix/h2-preface-before-redis

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Summary

The write path checked for HTTP/2 first only on ports 443 and 8443. On any other port the protocol detectors ran in order. is_redis_query accepts anything starting with an uppercase letter, so the client connection preface (PRI * HTTP/2.0…) was taken for a Redis command and the connection was cached as Redis.

Every write until the first server frame was read (which flips the connection to HTTP/2) was lost. Go and gRPC clients write their first request's headers before reading the server's SETTINGS. So on every new HTTP/2 connection to a port other than 443/8443, internal gRPC included, the first request's headers were lost.

Those headers carry most of the HPACK dynamic table's insertions: :authority, user-agent, content-type, the first :path. The decoder was missing them for the life of the connection, and every later request that referenced them failed to decode.

The preface is 24 fixed bytes and can't be mistaken for anything else, so it is now checked before the other detectors on every port.

Stacked on #359.

Engineering detail
  • TestProgramsLoad and TestGoTLSFdWalk pass on 5.10 and 6.1. sys_enter_sendmsg processes 178,069 verifier instructions.
  • The Redis inline-command check is still very loose (any uppercase first letter). Tightening it is left out of this change.

Local e2e: built agent images with and without this change, each with a local-only patch that dumps the HTTP/2 events the parser receives, and ran them in a local Docker VM (kernel 6.10) against a Go HTTP/2 client and server on port 9443. Without the change, the first HTTP/2 client event was a mid-stream write: the preface, the SETTINGS ACK and the first request chunk never reached the agent. With it, the first event is the preface and the whole client stream replays as aligned frames. In the same setup, with a Go net/http client making ~1.6M requests per 60s run: the parent image gave 1.5M–3M HPACK decode errors. This change together with #357 (decoder) and the stream-cap PR (#361) gave 0 errors in both runs, and every request the client made was decoded. This change alone does not remove the errors, because of the table-size-update cascade #357 fixes. go vet and the eBPF load tests pass.

The write path detected HTTP/2 first only on ports 443 and 8443. On any
other port the protocol detectors ran in order, and is_redis_query
accepts anything that starts with an uppercase letter, so the client
connection preface ("PRI * HTTP/2.0...") was taken for a Redis command
and the connection was cached as Redis. Every write until the first
server frame was read, which flips the connection to HTTP/2, was lost.

Go and gRPC clients write their first request's headers before reading
the server's SETTINGS, so those headers were lost on every new HTTP/2
connection to a port other than 443/8443, internal gRPC included. They
carry most of the HPACK dynamic table's insertions, so the decoder was
missing them for the life of the connection: every later request that
referenced them failed to decode.

The preface is unambiguous, so it is now checked before the other
detectors on every port.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the HTTP/2 detection logic in trace_enter_write to check for the client connection preface on any port, preventing connections from being misidentified as Redis. The review feedback suggests optimizing the logical condition to avoid redundant checks and save eBPF verifier and runtime instructions on the write hot path.

Comment thread ebpftracer/ebpf/l7/l7.c Outdated
Comment on lines +651 to +653
if (conn->dport != 53 && http2_detection_allowed(conn) &&
(is_likely_http2_port(conn->dport) || is_http2_client_preface(payload, size)) &&
looks_like_http2_frame(payload, size, METHOD_HTTP2_CLIENT_FRAMES)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since is_http2_client_preface(payload, size) being true guarantees that looks_like_http2_frame(payload, size, METHOD_HTTP2_CLIENT_FRAMES) will also return 1 (as it immediately checks and returns 1 on the preface), we can simplify the logical expression.

By rewriting the condition to is_http2_client_preface(payload, size) || (is_likely_http2_port(conn->dport) && looks_like_http2_frame(payload, size, METHOD_HTTP2_CLIENT_FRAMES)), we avoid:

  1. Calling is_http2_client_preface twice when the preface is present on a non-standard port.
  2. Calling looks_like_http2_frame entirely when the preface is matched.

This reduces redundant checks and saves eBPF verifier/runtime instructions on the write hot path.

        if (conn->dport != 53 && http2_detection_allowed(conn) &&
            (is_http2_client_preface(payload, size) ||
             (is_likely_http2_port(conn->dport) && looks_like_http2_frame(payload, size, METHOD_HTTP2_CLIENT_FRAMES)))) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in a6631d9: the preface check now short-circuits, and frame detection runs only on the 443/8443 path.

The write probe moved past the stack check, so ebpf.go is regenerated.
The preface check also short-circuits the frame detection it implies.
@mayankpande88

Copy link
Copy Markdown
Contributor Author

Folded into #364 with the rest of this stack, so it can be reviewed and merged as one unit (squash-only merges made each stacked merge conflict). The commit is unchanged there; its description and review thread here stay as the detailed write-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant