Skip to content

perf(ebpf): wake the proc_events reader per event instead of polling every 10 ms - #366

Merged
mayankpande88 merged 4 commits into
mainfrom
perf/proc-events-wakeup
Oct 6, 2026
Merged

mayankpande88 merged 4 commits into
mainfrom
perf/proc-events-wakeup

Conversation

@mayankpande88

@mayankpande88 mayankpande88 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Perf readers are created with WakeupEvents: 100: the kernel wakes a reader once a CPU's buffer holds 100 events, and below that the events wait for the reader's next deadline. Process events come a few at a time. So #353 cut the proc_events deadline from 100 ms to 10 ms, to act on an exec before the new program connects. Polling 100 times a second cost about 5 millicores on an idle node.

proc_events is now created with WakeupEvents: 1, so an exec wakes the reader at once. Because the kernel wakes it for every event, it has no read deadline at all; Close still interrupts a blocked read. The other readers keep their deadlines: 100 ms by default, 10 ms for tcp_connect_events.

This is the agent-CPU follow-up from the #353 review.

Engineering detail
  • Diagnosis: an idle CPU profile of the agent before and after fix(tls): count TLS capture losses and fix the gaps they exposed #353 shows the extra time entirely in the perf reader's epoll loop (runEventsReader → ReadInto → EpollWait): 0.57 s → 1.0 s per 90 s.

  • Scope: only proc_events changes. tcp_connect_events keeps its 10 ms deadline, because connects can be frequent enough that a wakeup per event would cost more than the polling.

  • CPU, from the agent's own cgroup: idle went from 22 to 13–15 millicores, against 18–19 before fix(tls): count TLS capture losses and fix the gaps they exposed #353. With ~3.3 short-lived Go CLI execs a second it went from 35 to 28–33.

  • Regression suite: 394 captured from 390 short-lived TLS clients (some make two requests; main: 388–390), same-path 131, wrapper+exec 21 (main: 17–18), no races.

Local e2e: built agent images from this branch and from main and ran each in a local Docker VM (kernel 6.10). I measured the agent's own cgroup CPU over a 180s idle window, then over 300s while a stripped Go CLI linking crypto/tls ran ~3.3 times a second. Idle: main 22, this branch 13–15 millicores. Under the CLI: main 35, this branch 28–33. Runs that overlapped my own builds in the same VM are excluded. The short-lived-process scenario suite is at parity or better, with no data races.

…every 10 ms

Perf readers are created with WakeupEvents 100: the kernel wakes a
reader once a CPU's buffer holds 100 events, and below that the events
wait for the reader's next deadline. Process events come a few at a
time, so to act on an exec before the new program connects, the
proc_events deadline was cut to 10 ms. Polling 100 times a second cost
~5 millicores on an idle node (agent CPU profile: the perf reader's
epoll loop went from 0.57 s to 1.0 s per 90 s).

proc_events is now created with WakeupEvents 1, so an exec wakes the
reader at once, and its deadline is back to the default 100 ms.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a wakeupEvents configuration for perfMap in ebpftracer/tracer.go to wake up the reader on every event for proc_events instead of polling, aiming to reduce idle CPU usage. However, feedback points out that because runEventsReader still defaults a zero readTimeout to 100 * time.Millisecond and sets a deadline on every iteration, the idle wakeups are not fully eliminated. It is recommended to update runEventsReader to only set a deadline if readTimeout > 0 and explicitly configure a readTimeout for other maps.

Comment thread ebpftracer/tracer.go
proc_events is created with WakeupEvents 1, but runEventsReader still
gave it the default 100 ms deadline, so an idle node woke its reader 10
times a second for nothing. A reader the kernel wakes for every event
now has no deadline; the others keep 100 ms unless set otherwise.
Close still interrupts a blocked Read.
@mayankpande88
mayankpande88 merged commit ae1f2d2 into main Oct 6, 2026
7 checks passed
@mayankpande88
mayankpande88 deleted the perf/proc-events-wakeup branch October 6, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants