Skip to content

fix(dns): stop labelling DNS queries with an earlier TCP connection's server - #393

Merged
mayankpande88 merged 2 commits into
mainfrom
fix/dns-actual-destination
Oct 10, 2026
Merged

mayankpande88 merged 2 commits into
mainfrom
fix/dns-actual-destination

Conversation

@mayankpande88

@mayankpande88 mayankpande88 commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

DNS series could carry the wrong actual_destination label. A query to a resolver was labelled with the server of some unrelated TCP connection the same container had made earlier, such as 198.51.100.80:8080 for a query sent to :53. Each wrong value added a new series for every queried domain, in both the counter and the histogram, and the count grew with agent uptime. On busy nodes this can push /metrics past a scraper's response size limit again, the same failure #372 fixed for the destination label.

This PR keeps a DNS query's actual destination as the address it was sent to.

Engineering detail

Cause. Since #372, DNS queries build their connection from their own socket tuple (connectionFromSocketInfo). That path looks up the socket's post-NAT destination with Tracer.ActualDestination(src), a read of the actual_destinations map. The map is written from conntrack in handle_ct, which returns early for anything but TCP. It is keyed by local IP:port only and is an LRU, so entries are never removed. A UDP DNS socket therefore has no entry of its own. Its lookup returns either nothing or the translation of an earlier TCP connection that happened to use the same local port number. UDP and TCP port allocation are independent, so with many connections a collision is routine.

Fix.

  • connectionFromSocketInfo takes a tcp argument. The DNS branch passes false and skips the lookup, so connectionKey falls back to the Cilium table (which matches the full TCP tuple) and then to the socket's destination.
  • createConnectionFromSocketInfo, which handles TCP connections whose open event was missed (Go TLS), passes true and is unchanged.
  • The socket tuple does not record whether a socket is UDP or TCP. DNS over TCP reaching this branch (not tracked from its connect) loses its translation. That case is rare, and the result is the address the query was sent to, not a wrong one.
  • Registry.actualDestination holds the lookup as a function (set to tracer.ActualDestination by NewRegistry) so a test can stand in for the kernel map.

Review follow-up (8edbd7c): dropped the nil check on the lookup. The test helper sets a lookup that finds nothing.

Test. TestDNSQueryTakesNoTCPTranslation stubs the lookup to return a TCP server for any local address. A DNS query must keep actual_destination="192.0.2.53:53", and a TCP connection built from its tuple must still take the stubbed translation. Mutation check: restoring the unconditional lookup fails the test with DNS actual destinations = map[198.51.100.80:8080:true].

CI-equivalent checks. In a Linux container with Go 1.26.5: gofmt, go vet ./..., golangci-lint v2.13.2 (0 issues), and go test ./... (including /containers) all pass.

Local e2e: in Docker Desktop (kernel 6.10, arm64), I ran the published 0.1.10-rc.2 image and a build of this branch as the agent. A Python client container opened 4,000 TCP connections to a server listening on 20 ports. It then sent 2,000 DNS queries to a dnsmasq container, each on a new UDP socket.

  • rc.2: 100 DNS series for 5 domains, with 21 distinct actual_destination values. 20 of them were the TCP server's ports (<server>:8001–:8020) on queries sent to <resolver>:53.
  • This branch, run three times (the first version of the change, the version before the review commit, and 8edbd7c): 5 series, one per domain, each with actual_destination="<resolver>:53". All 2,000 queries were counted, the client's TCP connect series still carry their 20 actual destinations, and there were no panics.

… server

DNS queries take their connection from their own socket tuple, and the
tuple path looked up the socket's post-NAT destination in the kernel's
translation table. That table is filled from conntrack for TCP connections
only and is keyed by local address alone. For a UDP query it could only
return the entry of an earlier TCP connection that had used the same local
port, so queries to one resolver came out with the actual destinations of
whatever servers the container had connected to. Each one added a set of
series for every queried domain, in the counter and the histogram, and the
series grew with uptime.

The DNS path no longer takes a translation, so a query's actual
destination is the address it was sent to. TCP connections built from
their tuple still take theirs. The lookup is held by the registry as a
function so a test can stand in for the kernel table.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that post-NAT destination lookups are only applied to TCP connections, preventing UDP-based DNS queries from incorrectly inheriting earlier TCP connection translations. This is implemented by adding a tcp boolean flag to connectionFromSocketInfo. The feedback suggests optimizing the hot path in connectionFromSocketInfo by removing the defensive nil check on c.registry.actualDestination and instead ensuring it is always initialized, including in test helpers.

Comment thread containers/container.go Outdated
…393 review)

NewRegistry always sets the lookup, so the check only served tests. The
test helper now sets a lookup that finds nothing.
@mayankpande88
mayankpande88 merged commit 1802462 into main Oct 10, 2026
7 checks passed
@mayankpande88
mayankpande88 deleted the fix/dns-actual-destination branch October 10, 2026 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants