Port upstream coroot-node-agent v1.29.0 → v1.36.2

[mayankpande88]

Tracking issue for porting coroot/coroot-node-agent v1.29.0 → v1.36.2 into this fork.

Where we are

• Everything up to upstream v1.28.3 is already here. Merge upstream coroot/coroot-node-agent changes #201 brought it in as a squash, so git doesn't know: git merge-base still points at 2025-06-03. That's why a plain git merge upstream/main conflicts everywhere.
• 49 upstream commits since v1.28.3. Each was cherry-picked on its own onto main and the conflict hunks counted, ignoring the generated ebpftracer/ebpf.go. The counts are in the tables below.
• We already have equivalents of two of them:
  • coroot/coroot-node-agent@7efe671 (OpenSSL fd capture from syscalls) is fix(ebpf): capture OpenSSL traffic regardless of BIO type and OpenSSL version #346 here, implemented with different BPF maps.
  • coroot/coroot-node-agent@91bb874: our Dockerfile already pins snapshot.debian.org.
• rustls, Java TLS and inbound L7 can't be cherry-picked. Upstream builds them on 7efe671's maps (ssl_pending, *_last_read_fd). Here they have to be ported onto fix(ebpf): capture OpenSSL traffic regardless of BIO type and OpenSSL version #346's ssl_write_pending / ssl_read_pending / ssl_fds.
• Upstream's Windows refactor (coroot/coroot-node-agent@34de3a0) moved files. It split flags/, node/ and gpu/ into *_linux.go / *_windows.go. Later upstream commits that touch flags/flags_linux.go or windows/* need their paths remapped by hand.
• The upstream log commits need logparser ≥ v1.4.2. Our logparser fork is 6 upstream commits behind.

How to work through it

• One PR per batch, in order.
• git cherry-pick -x so every commit message keeps the upstream SHA.
• After any .c change, regenerate with cd ebpftracer && make build.
• Each PR needs a local end-to-end run, and a rollout to a test node checked for up == 0 and duplicate-series gather errors before the next batch starts.
• Compare cumulative counters, not short windows.

Batches

B1: push path and service lifecycle (Go only)

• coroot/coroot-node-agent@273c780: don't get stuck on a bad spool chunk (1 hunk) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)
• coroot/coroot-node-agent@a5999af: remote-write with an explicit Content-Length (clean) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)
• coroot/coroot-node-agent@fa609da: --ca-file for TLS verification (2) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)
• coroot/coroot-node-agent@75d6656: systemd services disappearing after a restart (3) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)
• coroot/coroot-node-agent@3d898f9: stale container mounts producing duplicate disk metrics (2) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)
• coroot/coroot-node-agent@2d90d4a: multiple ephemeral port ranges (3) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)
• coroot/coroot-node-agent@5b2cb5f: recognise podruntime.slice (clean) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)
• coroot/coroot-node-agent@91fdf8d: php-fpm app type (clean) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)
• coroot/coroot-node-agent@1e261c3: --log-level (2) already covered by fix: log each agent message once, and skip untracked sockets before the map lookup #358
• coroot/coroot-node-agent@f76895d: report missing BPF tracing program types clearly (1) (fix: port upstream push-path and container-tracking fixes (B1 of #369) #370)

Done when:

• Remote-write to a store behind TLS signed by a private CA works with CA_FILE.
• An empty or corrupt file in WAL_DIR is discarded and sending resumes.
• systemctl restart <unit> keeps the unit's series continuous.

B2: cardinality guards

• coroot/coroot-node-agent@33c46ec: cap unique FQDNs per container in container_dns_requests_total; the overflow goes to ~other (4) (fix: cap DNS domain labels per container, add opt-in --min-container-age (B2 of #369) #371)
• coroot/coroot-node-agent@34ea61f: --min-container-age (3) (fix: cap DNS domain labels per container, add opt-in --min-container-age (B2 of #369) #371)

Decided: 50 FQDNs. --min-container-age ships at 0 (off), changed from upstream's 30s after review: with 30s, crash-looping pods at the maximum backoff and jobs under 30s never reported.

Done when: the series count per node is measured before and after, and containers living longer than 30s keep all their series.

B3: L7 parser correctness (eBPF)

• coroot/coroot-node-agent@2d8fb1b: RabbitMQ/ClickHouse protocol misclassification (5)
• coroot/coroot-node-agent@b53b38d: stronger ClickHouse detection (1)
• coroot/coroot-node-agent@a8a084f: unbounded allocations and lost queries in the ClickHouse parser (10)
• coroot/coroot-node-agent@556154b: bounded offset in the ClickHouse query parser (fixes program load) (2)
• coroot/coroot-node-agent@6d5addb: drop L7 events with unknown protocols (4)

Done when:

• The programs load on a 5.8 kernel and on a current kernel.
• Verified instruction counts stay within limits.
• A ClickHouse client and a RabbitMQ client are classified correctly.
• L7 drop and unknown counters don't regress over 24h.

B4: uprobe lifecycle (port; overlaps #349, #352, #354)

• coroot/coroot-node-agent@f8d437a: deduplicate uprobes (23)
• coroot/coroot-node-agent@2fdd10d: uprobe leak from a race between the instrument goroutine and Process.Close (9)
• coroot/coroot-node-agent@2c72586: register uprobes before reporting running processes (1)
• coroot/coroot-node-agent@44e3e8e: startup delay for Python GIL / Node.js event-loop instrumentation (2)
• coroot/coroot-node-agent@0ae7f5a: apply the startup delay to TLS uprobes too (8)

Decided: the TLS attach delay ships at 0. Upstream's 30s would drop the first 30s of every process's TLS traffic, and all of it for processes that live less than 30s. Measure attach CPU with #352 in place before changing that. Upstream uses a single flag for both delays, so keeping Python/Node at 30s with TLS at 0 needs a separate flag; settle that in the PR.

Done when:

• The uprobe count on a test node stays flat over 24h.
• TLS-attach CPU is no worse than before.
• Plaintext captured for long-lived processes doesn't drop.

B5: TLS runtimes (port onto the #346 hooks)

• coroot/coroot-node-agent@913e9d4: rustls plaintext capture (7, plus rework onto our maps)
• coroot/coroot-node-agent@579eef8: Java TLS through a Java agent + uprobes (11, plus jar and native assets)

The Java agent is injected into running JVMs. It stays opt-in, off by default.

Done when:

• A rustls client and a Java HttpClient calling HTTPS produce HTTP metrics with the right status, and spans with paths.
• With the Java agent enabled, the JVM stays stable under load.

B6: inbound L7 (port; depends on B3–B5)

• coroot/coroot-node-agent@fa6f500: instrument inbound L7 requests (46 hunks, plus the B5 files)

It adds container_*_inbound_* metrics labelled status (plus method for RabbitMQ/NATS). It has to be re-checked against our variable-length ring records (#356) and the HTTP/2 resync and stream-cap changes (#364, #365).

Done when:

• A request from an uninstrumented client produces container_http_inbound_requests_total.
• With both sides instrumented, server-side MySQL, Redis and Postgres counts match client-side counts within 1%.
• Ring-buffer drops don't regress.
• The programs still load on 5.8.

B7: logs (logparser fork synced first)

• Sync the 6 missing upstream logparser commits into our logparser fork
• coroot/coroot-node-agent@cd65ce5: retry opening a container log file that doesn't exist yet (1)
• coroot/coroot-node-agent@e4be150: find the log file when the open() fd is already gone (4)
• coroot/coroot-node-agent@ab9b049: rate-limit log pattern extraction under log storms (8; drop the windows/ parts)
• coroot/coroot-node-agent@e1cd714: extract attributes from JSON logs (6; drop the windows/ parts)
• coroot/coroot-node-agent@18cfe3c: OTLP trace context from trace_id/span_id in JSON logs (2)

Done when:

• A log storm into one container doesn't push the agent's CPU past the cap.
• JSON log lines become OTLP records with attributes and trace context.

B8: process start and boot time

• coroot/coroot-node-agent@27bc75d: process start time from /proc/<pid>/stat instead of taskstats (1)
• coroot/coroot-node-agent@8c14700 and coroot/coroot-node-agent@a9fd101: read the host boot time once (2 + 3)

Done when: process start times are unchanged and the taskstats calls are gone from the CPU profile.

After the port

• Once every commit above is ported or skipped, open a PR with git merge -s ours <upstream tag>. It changes no files; it gives git the right merge base, so the next sync starts from v1.36 instead of 2025-06-03.

Not porting (for now)

Commits	Reason
34de3a0, 481feb0, and the windows/* parts of later commits	Windows support is out of scope
e5818fc, e73d36c, b9fc7fd, 675758e, 835ae85, 0c934cd, 8992a5b	JVM async-profiler and Go heap profiling: revisit with the profiling work. The async-profiler is also injected into JVMs
f6813ff	docker → moby client: do it when the docker client deprecation requires it
de67715, ae3e2e6	group AWS/GCP endpoints by FQDN: overlaps the FQDN resolution in #313–#315; compare naming first
f97a3f8	ping timestamping: the pinger is off by default here
80d4530	RHEL 7 / kernel 3.10: our L7 path needs BPF ring buffers (5.8+, #363), so this needs a perf-buffer fallback design. First test whether RHEL 8's backported ring buffer loads our programs
2de271a	removes the Vagrant test setup, which we don't have
7efe671, 91bb874	already covered (#346, Dockerfile)

[mayankpande88]

B1 is in #370. Two changes to the plan:

• coroot/coroot-node-agent@1e261c3 is dropped. fix: log each agent message once, and skip untracked sockets before the map lookup #358 already logs each message once.
• The createdAt fallback from coroot/coroot-node-agent@75d6656 moves to B2. It only applies with --min-container-age.

75d6656 also turned out to be mostly covered here already: #353 registers processes on exec, listen and connect events. B1 takes the rest of it: registration for every event type, and a locked lookup.

[mayankpande88]

B2 is in #371, stacked on #370.

• coroot/coroot-node-agent@33c46ec is re-implemented in L7Stats, because this fork records DNS with destination labels.
• coroot/coroot-node-agent@34ea61f ships with the 30s default. It now includes the createdAt fallback held back from 75d6656.
• A pid is treated as exited only when /proc/<pid> is gone.

Also found during the e2e (not part of B2): on every build, a burst of new systemd units started within about a minute of the agent starting can go undetected, 1–2 of 10 per burst.

[mayankpande88]

B2 change after review: --min-container-age ships with a default of 0 (off), not upstream's 30s. A pod in CrashLoopBackOff at the 5-minute maximum backoff is removed between restarts and comes back as a new container each time, so with 30s it never reported. Short jobs were hidden too. The flag stays available for installs that want it.

[mayankpande88]

B3 is in #377: coroot/coroot-node-agent@a8a084f (new ClickHouse parser), b53b38d and 556154b (stricter eBPF detection), plus regenerated ebpf.go.

Two B3 commits aren't needed here:

• 2d8fb1b, eBPF half: this fork already classifies every AMQP frame on 5672 as RabbitMQ before the port-gated ClickHouse check.
• 6d5addb: L7Stats already ignores protocols that have no metrics.

In the e2e, main lost the text of every query that carried settings (empty db.statement); #377 keeps it. The programs load on 5.10 and 6.1.

[mayankpande88]

B4 is split.

B4a is #378.

• 2c72586 is ported: uprobe programs are registered before anything is attached.
• 44e3e8e is in with --instrumentation-delay defaulting to 0. The measured saving was about 2.6 millicores at 300 short-lived Python processes a minute, because Node.js tracing is already opt-in here.
• 2fdd10d isn't needed: fix(tls): count TLS capture losses and fix the gaps they exposed #353 already closes links that arrive after Close.

B4b, still to do: f8d437a (uprobe deduplication) and 0ae7f5a (delay for TLS uprobes). This reworks TLS attach here, so it needs a spec first.

• Design:
  • Attach each uprobe set once per binary (dev + inode), refcounted per process, with no pid filter. Today every attach carries a pid filter (elf.go, symbol_cache.go).
  • go_tls_offsets_map stays keyed by TGID. Each new process of an already-attached binary still needs its entry written, but no attach.
  • libssl.so is shared by many binaries, which makes it the biggest win.
  • The guarantees from fix(tls): count TLS capture losses and fix the gaps they exposed #353 (attach before the first connect, exec re-check) and feat(tls): attribute TLS losses per container and find sockets behind wrapped Go connections #354 (wrapper walk) must hold.
• Measure first: TLS-attach CPU and capture rate, using the short-lived-client scenario (about 390 clients, baseline 388–390 captured) and an attach-share CPU profile on a busy node.
• Done when:
  • the short-lived capture rate is at or above baseline;
  • the number of uprobe links scales with distinct binaries, not processes;
  • TLS-attach CPU is lower;
  • the link count returns to baseline after process churn (no leaks);
  • long-lived TLS capture is unchanged;
  • the programs load on 5.10 and 6.1.
• 0ae7f5a ships with a TLS delay of 0 unless the measurement justifies more.

[mayankpande88]

B8 is in #379: 27bc75d, 8c14700 and a9fd101, plus a test for GetStartTime. Process registration no longer depends on taskstats. At about 100 process starts per second, the agent used 3667 ms of CPU over 60s, against 4251 ms on main.