Why do Kubernetes Pods send failed DNS lookups before external API calls?

Your checkout Pod makes one call to api.payments-provider.com, yet CoreDNS logs show a burst of NXDOMAIN answers for names nobody asked for. Each failed lookup adds outbound latency, and only the last query succeeds. The cause isn't the API or CoreDNS. It's the resolv.conf file inside the Pod.

Inside the checkout Pod, ClusterFirst DNS typically writes three lines. The nameserver is the cluster DNS Service, 10.96.0.10. The search list starts with the Pod's namespace, then broader cluster domains, and here a node domain. The last line, ndots:5, is the trigger. It sets how many dots a name needs before it's tried as-is.

The app asks the resolver for api.payments-provider.com. That name has two dots, which is below ndots five, so the resolver treats it as a short name. It appends each search suffix first, usually asking for both A and AAAA records. CoreDNS answers NXDOMAIN for checkout.svc, then svc.cluster.local, then cluster.local. Every one of those lookups is wasted.

The last suffix, the node domain, fails the same way. CoreDNS answers NXDOMAIN again, and that domain can vary by environment. Only now does the resolver ask for api.payments-provider.com as-is. CoreDNS forwards it to upstream DNS, which returns the IP address. The resolver hands that IP back to the app, and the call can finally proceed.

Add up the bars. Cluster suffixes cost 6 queries, the node suffix costs 2, and that's 8 failing lookups. Only then do the 2 queries for the real name run. That's illustrative, assuming A and AAAA queries. Caching and connection reuse change how often this repeats.

The resolver counts the dots in the queried name and compares that to ndots. Fewer dots means search first. Enough dots means as-is first. Kubernetes picks 5 so short names like orders or orders.payments expand through the search list to Services. The cost is that external names like our payments API pay for it.

This is a cost problem, not a correctness problem. The extra NXDOMAIN lookups add load and latency, but the right answer still arrives. How much you pay depends on glibc, musl on Alpine, or your language's resolver, so observe your own workload.

You have two client-side levers. Option A adds a trailing dot, so the resolver treats the name as fully qualified and skips the search list. But some HTTP clients, TLS SNI, or Host header checks mishandle it, so test. Option B sets ndots to 2 for this Pod, which fixes the API lookup but changes multi-dot internal names.

Caches can absorb repeat lookups at two places. An app cache removes queries entirely, but if it ignores TTLs, it can hold stale IPs. NodeLocal DNS doesn't skip the suffixed queries. It answers them nearby, NXDOMAINs included, so latency drops and CoreDNS sees less load, while the Pod resolver behaves the same.

Before you change anything, confirm the problem. Start by reading resolv.conf inside the checkout Pod. Then check CoreDNS logs or NXDOMAIN metrics for suffixed names. Record a baseline for DNS and call latency, change one thing, and re-measure. That comparison tells you whether the mitigation actually helped.

With ndots:5, names under five dots try search suffixes first. Before choosing trailing dots, ndots, or caching, confirm the NXDOMAIN volume and latency impact.