Skip to content

fix(ebpf): flag events lost to a failed copy, and mark TLS on the tracked connection - #367

Merged
blue4209211 merged 2 commits into
mainfrom
fix/l7-lost-gaps
Oct 6, 2026
Merged

blue4209211 merged 2 commits into
mainfrom
fix/l7-lost-gaps

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Summary

Two gaps the review of #365 found in the lost-event handling:

  • Failed copies weren't flagged. An event dropped because bpf_probe_read of the user buffer failed was dropped before send_event, so connection.l7_lost wasn't set and the HTTP/2 parser couldn't see the gap. COPY_PAYLOAD_RINGBUF now records the loss on the connection: on the written side in the write path, on the read side in the read path.
  • The TLS mark could miss the tracked connection. On a TLS write to an untracked connection, mark_tls ran on the stack copy before the code switched to the map entry. If another CPU had inserted that entry first (the TCP connect path does, with tls = 0), the entry never got the mark and its socket-level ciphertext wasn't skipped. The mark is now applied again after the switch.
Engineering detail
  • The macro now takes the connection and the loss bit explicitly. All 11 call sites are converted, 6 on the write path and 5 on the read path.
  • The eBPF programs load on 5.10 and 6.1. sys_enter_sendmsg processes 207,751 instructions.
  • Not done here: the non-atomic l7_lost race. The fix the review proposes is a u32 counter per direction, incremented with XADD. That grows struct connection past 32 bytes, so it waits for dev numbers on events_lost to show the race matters.

Local e2e: built the agent from this branch and ran it in a local Docker VM (kernel 6.10). A Go net/http HTTP/2 client: every request decoded (1,637,696), 0 decode errors. Bulk downloads plus streaming over one server: every stream completed. The short-lived-process suite, including the OpenSSL clients that use the untracked-connection path, is at parity: 386 of 390 short-lived clients captured, Python 19/20, no data races. A failed user-buffer copy can't be provoked locally, so that path is covered by the verifier load and code review only.

…cked connection

Two gaps left after the lost-event flag (from review of #365):

- An event dropped because bpf_probe_read of the user buffer failed was
  dropped before send_event, so l7_lost was not set and the parser could
  not see the gap. COPY_PAYLOAD_RINGBUF now records the loss on the
  connection, on the written or read side by where it runs.
- On a TLS write to an untracked connection, mark_tls ran on the stack
  copy before the code switched to the map entry. If another CPU had
  inserted that entry first (the TCP connect path does, with tls 0), it
  never got the mark and its socket-level ciphertext was not skipped. The
  mark is now applied again after the switch.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the COPY_PAYLOAD_RINGBUF macro in l7.c to record payload copy failures on the connection state, using new constants L7_LOST_WRITES and L7_LOST_READS instead of magic numbers. It also adds a call to mark_tls(conn) in trace_enter_write when a connection is retrieved from the active connections map. The review feedback points out that the macro parameter e is unused in COPY_PAYLOAD_RINGBUF and suggests removing it to clean up the macro definition and its call sites.

Comment thread ebpftracer/ebpf/l7/l7.c Outdated
The programs are unchanged (same instruction counts); ebpf.go differs
only in the BTF line info that carries the edited source lines.
@blue4209211
blue4209211 merged commit ab849e9 into main Oct 6, 2026
7 checks passed
@blue4209211
blue4209211 deleted the fix/l7-lost-gaps branch October 6, 2026 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants