Skip to content

fix: port upstream push-path and container-tracking fixes (B1 of #369) - #370

Merged
blue4209211 merged 10 commits into
mainfrom
port/upstream-b1-push-lifecycle
Oct 7, 2026
Merged

blue4209211 merged 10 commits into
mainfrom
port/upstream-b1-push-lifecycle

Conversation

@mayankpande88

@mayankpande88 mayankpande88 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

First batch of #369: nine commits from upstream coroot-node-agent (v1.29–v1.36), cherry-picked with -x. They make the agent more robust when it pushes data out, and fix how it tracks containers and systemd units.

Upstream commit Author Change
coroot/coroot-node-agent@273c780 Nikolay Sivko Remote-write no longer gets stuck on one spool chunk. The agent discards an empty chunk, and drops a chunk the receiver rejects with 400 or 413 instead of retrying it forever. Spool files are fsynced.
coroot/coroot-node-agent@a5999af Nikolay Sivko Remote-write requests carry a Content-Length instead of chunked encoding.
coroot/coroot-node-agent@fa609da Vishnu Kvs New --ca-file / CA_FILE: trust a private CA for remote-write, traces, logs and profiles.
coroot/coroot-node-agent@3d898f9 Nikolay Sivko A mount that leaves a container's mount namespace stops being reported. Before, its series stayed and showed whatever filesystem was now at that path.
coroot/coroot-node-agent@2d90d4a Nikolay Sivko --ephemeral-port-range accepts several ranges, and port 50051 (gRPC) is never treated as ephemeral.
coroot/coroot-node-agent@75d6656 Nikolay Sivko Every pid mapped to a container is registered as one of its processes.
coroot/coroot-node-agent@5b2cb5f Alexandre Proulx podruntime.slice is recognised as a systemd slice.
coroot/coroot-node-agent@91fdf8d Nikolay Sivko php-fpm application type.
coroot/coroot-node-agent@f76895d KR Ravindra Clear error when the kernel lacks kprobe or tracepoint BPF programs (CONFIG_BPF_EVENTS).

coroot/coroot-node-agent@1e261c3 (log each message once) isn't included: #358 already does the same here.

There's one more commit of our own, 2beb65f, from review. The send loop and the spool truncation both delete spool files. A file one of them deleted after the other had listed it used to make the send loop back off for 5s–1m, or make truncation fail and drop the payload being spooled. Both now skip a file that is already gone.

Engineering detail

Changes from upstream:

  • 3d898f9: upstream rewrites c.mounts in getMounts without taking a lock. In this fork, onFileOpen writes c.mounts under c.lock from the event goroutine, so the unchanged commit would cause concurrent map writes. Here the cleanup runs under c.lock, and the statfs loop works on a copy.
  • 75d6656:
    • This fork already had ensureProcess (fix(tls): count TLS capture losses and fix the gaps they exposed #353), called on exec, listen and connect events, so the restart case upstream fixed was already handled. What the commit adds here is registration inside getOrCreateContainer for every event type, file opens included.
    • The lookup in ensureProcess now holds c.lock.
    • Process-start events keep calling onProcessStart, because it replaces a process whose pid was reused.
    • Upstream's createdAt fallback belongs to --min-container-age and will come with that flag in the next batch.
  • fa609da: this fork sets OTLP TLS options only for https endpoints, so that branch now uses the shared TLS config.
  • 2d90d4a: only the flag help text changes; the fork keeps its own flag style.

Behaviour changes:

  • A remote-write 400 or 413 now drops that chunk instead of blocking every chunk behind it.
  • Connections to port 50051 are always tracked.

CI: gofmt, goimports, vet, golangci-lint, go test (excluding /containers) and the build all pass in a Linux container with Go 1.26.5.

2beb65f: CI checks pass. It covers a timing race, so the local e2e below didn't exercise it.

Squash merge: the commit list and the (cherry picked from commit …) lines carry the upstream attribution.

Local e2e: I built agent binaries from this branch and from main and ran them as systemd services on a local Debian 12 VM (kernel 6.1, systemd 252). They pushed to a local VictoriaMetrics and OpenTelemetry collector behind TLS signed by a private CA.

  • Private CA: with CA_FILE, this branch delivered metrics, traces and logs. Main failed with "certificate signed by unknown authority".
  • Content-Length: this branch's remote-write requests carried Content-Length; main's were chunked.
  • Spool: an empty chunk and a malformed chunk were queued ahead of four real ones. This branch discarded the empty one, dropped the malformed one on 400, and drained the rest. Main retried the malformed chunk with growing backoff while 10 chunks queued behind it.
  • Stale mount: after a lazy unmount of a loop-device volume a service was writing to, this branch stopped reporting that path. Main kept reporting it, with the root filesystem's size and usage.
  • Port ranges: with EPHEMERAL_PORT_RANGE=1024-23768,30000-65535, this branch skipped connections to :8080 and kept :25000.
  • Restart: a systemd unit restarted with a new main PID stayed in both builds' metrics past the 10-minute GC window.
  • Logs: neither agent logged an error or a panic.

Alegrowin and others added 9 commits October 7, 2026 16:12
(cherry picked from commit 5b2cb5f9f880c89ab3ac7983c0d6afa3643e47ad)
(cherry picked from commit 91fdf8d97d177b2b380ba258a95869f00bf0b0ef)
(cherry picked from commit fa609dac7bce1a06b224dcf8aaeff03f8525e040)
(cherry picked from commit a5999aff838846b8f5a460b141e534e16c89e204)
(cherry picked from commit 3d898f9b37b5972bc76447224d4d850fe888c135)

Fork adaptation: upstream updates c.mounts in getMounts without a lock;
here onFileOpen writes c.mounts under c.lock from the event goroutine, so
the reconcile runs under c.lock and the statfs loop uses a snapshot.
(cherry picked from commit 75d6656426ff5139443cfb2c9c06c3622ce013c4)

Fork adaptation: only the process-registration half is taken here. The
fork already had ensureProcess (#353); its lookup now holds c.lock. Start
events keep calling onProcessStart, which replaces a reused pid. The
createdAt fallback for --min-container-age comes with that flag (B2).
(cherry picked from commit 2d90d4abb42df1dd2a8b8a8b8b060e04fcb87a30)
(cherry picked from commit 273c780a4eea756467de15c6b3d6fc4e3fe56cd6)
On kernels built without CONFIG_BPF_EVENTS (e.g. NVIDIA JetPack 5 / L4T
5.10 with CONFIG_KPROBES=n), BPF_PROG_TYPE_TRACEPOINT and
BPF_PROG_TYPE_KPROBE are compiled out and bpf(BPF_PROG_LOAD) fails with a
bare EINVAL. The agent then exits with

  failed to load collection: program sched_process_exit: load program: invalid argument

where the program name is whichever program happened to load first, and
nothing points at the kernel configuration.

Probe TracePoint and Kprobe support with cilium/ebpf/features before
loading the collection and fail with an error naming the missing kernel
option. Only a conclusive ebpf.ErrNotSupported is reported; any other
probe result falls through to the real collection load so existing error
paths are unchanged.

Document the CONFIG_BPF_EVENTS requirement in the README.

(cherry picked from commit f76895d48f7ee98da2224376140351e057c6c336)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several enhancements and bug fixes to the node agent. Key changes include adding support for podruntime.slice cgroups, allowing multiple port ranges in the ephemeral port filter, updating PHP command regex to support php-fpm, and refactoring container mount tracking to prevent duplicate volume reporting. Additionally, a new --ca-file flag is introduced to support custom CA certificates across telemetry exporters, and the Prometheus remote writer is improved to read spool files into memory (avoiding chunked encoding issues), sync files to disk, and handle rejected payloads. Feedback on the changes highlights two issues: a potential nil pointer dereference in common/api.go if x509.SystemCertPool() returns a nil pool, and a race condition in prom/remote_writer.go where deleted spool files can trigger an unnecessary 5-second backoff delay instead of being skipped immediately.

Comment thread prom/remote_writer.go
Comment thread common/api.go
Comment thread common/api.go Dismissed
sendLoop and truncateSpoolIfNeeded (scrape goroutine) both remove spool
files. A file removed between listing and use made sendLoop back off
(5s to 1m) instead of moving to the next file, and made
truncateSpoolIfNeeded return an error, so writeToSpool dropped the
payload it was about to spool. Both now skip a file that is gone.
@blue4209211
blue4209211 merged commit 1131289 into main Oct 7, 2026
7 checks passed
@blue4209211
blue4209211 deleted the port/upstream-b1-push-lifecycle branch October 7, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants