Repository navigation
fix(dns): stop labelling DNS queries with an earlier TCP connection's server - #393
Merged
Merged
Conversation
… server DNS queries take their connection from their own socket tuple, and the tuple path looked up the socket's post-NAT destination in the kernel's translation table. That table is filled from conntrack for TCP connections only and is keyed by local address alone. For a UDP query it could only return the entry of an earlier TCP connection that had used the same local port, so queries to one resolver came out with the actual destinations of whatever servers the container had connected to. Each one added a set of series for every queried domain, in the counter and the histogram, and the series grew with uptime. The DNS path no longer takes a translation, so a query's actual destination is the address it was sent to. TCP connections built from their tuple still take theirs. The lookup is held by the registry as a function so a test can stand in for the kernel table.
There was a problem hiding this comment.
Code Review
This pull request ensures that post-NAT destination lookups are only applied to TCP connections, preventing UDP-based DNS queries from incorrectly inheriting earlier TCP connection translations. This is implemented by adding a tcp boolean flag to connectionFromSocketInfo. The feedback suggests optimizing the hot path in connectionFromSocketInfo by removing the defensive nil check on c.registry.actualDestination and instead ensuring it is always initialized, including in test helpers.
…393 review) NewRegistry always sets the lookup, so the check only served tests. The test helper now sets a lookup that finds nothing.
blue4209211
approved these changes
Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DNS series could carry the wrong
actual_destinationlabel. A query to a resolver was labelled with the server of some unrelated TCP connection the same container had made earlier, such as198.51.100.80:8080for a query sent to:53. Each wrong value added a new series for every queried domain, in both the counter and the histogram, and the count grew with agent uptime. On busy nodes this can push/metricspast a scraper's response size limit again, the same failure #372 fixed for thedestinationlabel.This PR keeps a DNS query's actual destination as the address it was sent to.
Engineering detail
Cause. Since #372, DNS queries build their connection from their own socket tuple (
connectionFromSocketInfo). That path looks up the socket's post-NAT destination withTracer.ActualDestination(src), a read of theactual_destinationsmap. The map is written from conntrack inhandle_ct, which returns early for anything but TCP. It is keyed by local IP:port only and is an LRU, so entries are never removed. A UDP DNS socket therefore has no entry of its own. Its lookup returns either nothing or the translation of an earlier TCP connection that happened to use the same local port number. UDP and TCP port allocation are independent, so with many connections a collision is routine.Fix.
connectionFromSocketInfotakes atcpargument. The DNS branch passesfalseand skips the lookup, soconnectionKeyfalls back to the Cilium table (which matches the full TCP tuple) and then to the socket's destination.createConnectionFromSocketInfo, which handles TCP connections whose open event was missed (Go TLS), passestrueand is unchanged.Registry.actualDestinationholds the lookup as a function (set totracer.ActualDestinationbyNewRegistry) so a test can stand in for the kernel map.Review follow-up (8edbd7c): dropped the nil check on the lookup. The test helper sets a lookup that finds nothing.
Test.
TestDNSQueryTakesNoTCPTranslationstubs the lookup to return a TCP server for any local address. A DNS query must keepactual_destination="192.0.2.53:53", and a TCP connection built from its tuple must still take the stubbed translation. Mutation check: restoring the unconditional lookup fails the test withDNS actual destinations = map[198.51.100.80:8080:true].CI-equivalent checks. In a Linux container with Go 1.26.5: gofmt,
go vet ./..., golangci-lint v2.13.2 (0 issues), andgo test ./...(including/containers) all pass.Local e2e: in Docker Desktop (kernel 6.10, arm64), I ran the published 0.1.10-rc.2 image and a build of this branch as the agent. A Python client container opened 4,000 TCP connections to a server listening on 20 ports. It then sent 2,000 DNS queries to a dnsmasq container, each on a new UDP socket.
actual_destinationvalues. 20 of them were the TCP server's ports (<server>:8001–:8020) on queries sent to<resolver>:53.actual_destination="<resolver>:53". All 2,000 queries were counted, the client's TCP connect series still carry their 20 actual destinations, and there were no panics.