Repository navigation
fix: port upstream push-path and container-tracking fixes (B1 of #369) - #370
Conversation
(cherry picked from commit 5b2cb5f9f880c89ab3ac7983c0d6afa3643e47ad)
(cherry picked from commit 91fdf8d97d177b2b380ba258a95869f00bf0b0ef)
(cherry picked from commit fa609dac7bce1a06b224dcf8aaeff03f8525e040)
(cherry picked from commit a5999aff838846b8f5a460b141e534e16c89e204)
(cherry picked from commit 3d898f9b37b5972bc76447224d4d850fe888c135) Fork adaptation: upstream updates c.mounts in getMounts without a lock; here onFileOpen writes c.mounts under c.lock from the event goroutine, so the reconcile runs under c.lock and the statfs loop uses a snapshot.
(cherry picked from commit 75d6656426ff5139443cfb2c9c06c3622ce013c4) Fork adaptation: only the process-registration half is taken here. The fork already had ensureProcess (#353); its lookup now holds c.lock. Start events keep calling onProcessStart, which replaces a reused pid. The createdAt fallback for --min-container-age comes with that flag (B2).
(cherry picked from commit 2d90d4abb42df1dd2a8b8a8b8b060e04fcb87a30)
(cherry picked from commit 273c780a4eea756467de15c6b3d6fc4e3fe56cd6)
On kernels built without CONFIG_BPF_EVENTS (e.g. NVIDIA JetPack 5 / L4T 5.10 with CONFIG_KPROBES=n), BPF_PROG_TYPE_TRACEPOINT and BPF_PROG_TYPE_KPROBE are compiled out and bpf(BPF_PROG_LOAD) fails with a bare EINVAL. The agent then exits with failed to load collection: program sched_process_exit: load program: invalid argument where the program name is whichever program happened to load first, and nothing points at the kernel configuration. Probe TracePoint and Kprobe support with cilium/ebpf/features before loading the collection and fail with an error naming the missing kernel option. Only a conclusive ebpf.ErrNotSupported is reported; any other probe result falls through to the real collection load so existing error paths are unchanged. Document the CONFIG_BPF_EVENTS requirement in the README. (cherry picked from commit f76895d48f7ee98da2224376140351e057c6c336)
There was a problem hiding this comment.
Code Review
This pull request introduces several enhancements and bug fixes to the node agent. Key changes include adding support for podruntime.slice cgroups, allowing multiple port ranges in the ephemeral port filter, updating PHP command regex to support php-fpm, and refactoring container mount tracking to prevent duplicate volume reporting. Additionally, a new --ca-file flag is introduced to support custom CA certificates across telemetry exporters, and the Prometheus remote writer is improved to read spool files into memory (avoiding chunked encoding issues), sync files to disk, and handle rejected payloads. Feedback on the changes highlights two issues: a potential nil pointer dereference in common/api.go if x509.SystemCertPool() returns a nil pool, and a race condition in prom/remote_writer.go where deleted spool files can trigger an unnecessary 5-second backoff delay instead of being skipped immediately.
sendLoop and truncateSpoolIfNeeded (scrape goroutine) both remove spool files. A file removed between listing and use made sendLoop back off (5s to 1m) instead of moving to the next file, and made truncateSpoolIfNeeded return an error, so writeToSpool dropped the payload it was about to spool. Both now skip a file that is gone.
Summary
First batch of #369: nine commits from upstream coroot-node-agent (v1.29–v1.36), cherry-picked with
-x. They make the agent more robust when it pushes data out, and fix how it tracks containers and systemd units.Content-Lengthinstead of chunked encoding.--ca-file/CA_FILE: trust a private CA for remote-write, traces, logs and profiles.--ephemeral-port-rangeaccepts several ranges, and port 50051 (gRPC) is never treated as ephemeral.podruntime.sliceis recognised as a systemd slice.CONFIG_BPF_EVENTS).coroot/coroot-node-agent@1e261c3 (log each message once) isn't included: #358 already does the same here.
There's one more commit of our own, 2beb65f, from review. The send loop and the spool truncation both delete spool files. A file one of them deleted after the other had listed it used to make the send loop back off for 5s–1m, or make truncation fail and drop the payload being spooled. Both now skip a file that is already gone.
Engineering detail
Changes from upstream:
c.mountsingetMountswithout taking a lock. In this fork,onFileOpenwritesc.mountsunderc.lockfrom the event goroutine, so the unchanged commit would cause concurrent map writes. Here the cleanup runs underc.lock, and the statfs loop works on a copy.ensureProcess(fix(tls): count TLS capture losses and fix the gaps they exposed #353), called on exec, listen and connect events, so the restart case upstream fixed was already handled. What the commit adds here is registration insidegetOrCreateContainerfor every event type, file opens included.ensureProcessnow holdsc.lock.onProcessStart, because it replaces a process whose pid was reused.createdAtfallback belongs to--min-container-ageand will come with that flag in the next batch.Behaviour changes:
CI: gofmt, goimports, vet, golangci-lint,
go test(excluding/containers) and the build all pass in a Linux container with Go 1.26.5.2beb65f: CI checks pass. It covers a timing race, so the local e2e below didn't exercise it.
Squash merge: the commit list and the
(cherry picked from commit …)lines carry the upstream attribution.Local e2e: I built agent binaries from this branch and from main and ran them as systemd services on a local Debian 12 VM (kernel 6.1, systemd 252). They pushed to a local VictoriaMetrics and OpenTelemetry collector behind TLS signed by a private CA.
CA_FILE, this branch delivered metrics, traces and logs. Main failed with "certificate signed by unknown authority".Content-Length; main's were chunked.EPHEMERAL_PORT_RANGE=1024-23768,30000-65535, this branch skipped connections to :8080 and kept :25000.