Understanding DNS in Kubernetes

Povilas

Understanding DNS in Kubernetes

Introduction

DNS is an integral part of the Kubernetes Control Plane. It allows you to use human-readable domain names instead of IP addresses. In Kubernetes, DNS is used for service discovery, allowing applications to connect to other pods directly or via a service load balancer.

Usually, In Kubernetes, CoreDNS serves as the DNS provider. It runs as a regular Deployment inside the Kubernetes cluster. CoreDNS is written in Go and has a plugin-based model controlled by a configuration file.

In this post, we will cover the following:

Overview of DNS Resolution and CoreDNS, the default DNS provider in Kubernetes.
Kubernetes DNS policies, such as ClusterFirst, Default, and None, and their effects on pod DNS configurations.
Differences between The GNU C Library (glibc) and musl libraries.

DNS resolution

Applications typically interact with a DNS server through the C standard library (libc) using the getaddrinfo function call. You can control how DNS resolution is handled by configuring /etc/resolv.conf file on Linux. One important setting is nameserver, which specifies the IP address of your DNS server. In the Kubernetes cluster, by default, when running a pod with default DNS policy (ClusterFirst) nameserver will have a 10.96.0.10 value, which is an IP address of kube-dns Service that load balances TCP and UDP traffic to CoreDNS. Note: some providers like k3s use different ranges for service IPs, the specified IP range comes from service-cluster-ip-range flag in kube-api-server component. The k3s uses 10.43.0.0/16 range, so the kube-dns service IP is at 10.43.0.10.

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
  name: kube-dns
  namespace: kube-system
spec:
  clusterIP: 10.96.0.10
  clusterIPs:
  - 10.96.0.10
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  - name: metrics
    port: 9153
    protocol: TCP
    targetPort: 9153
  selector:
    k8s-app: kube-dns
  sessionAffinity: None
  type: ClusterIP

Note: Kubernetes kube-dns Service definition.

When using the default ClusterFirst DNS Policy, Kubelet will generate the following /etc/resolv.conf:

search <pod-namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The search option provides a search list of host-name lookups. DNS queries with fewer dots than specified in the ndots option will be attempted using each element of the search field until a match is found. For example, your Pod queries DNS for app.kube-system. Then the DNS client checks “1 dot is less than 5 dots” and attempts to lookup:

app.kube-system.<pod-namespace>.cluster.local.
app.kube-system.svc.cluster.local.
app.kube-system.cluster.local.
app.kube-system.

Note that if your query ends with a dot, it’s considered a Fully Qualified Domain Name. In this case, the DNS client will resolve the DNS entry exactly as it is without using the elements in the search option.

It’s important to note that if you do a DNS lookup to an external domain without a dot at the end, such as google.com, you will always send 4 queries to your CoreDNS server. Only the last query will be google.com, which CoreDNS will send to your node’s nameserver.

This is how everything works with the default ClusterFirst Pod DNS policy. But what about the other options?

Kubernetes Pod DNS Policy

Kubernetes additionally supports the following DNS Policies:

Default—The Pod inherits the /etc/resolv.conf from the node it runs on. For instance, a CoreDNS Pod typically uses this policy to forward external domain name requests to the node’s nameserver.
ClusterFirstWithHostNet—This policy is for pods with host network enabled, applying ClusterFirst behavior. Otherwise, these pods default to the Default policy.
None: This policy allows custom DNS configurations on the pod spec. Here’s an example:

apiVersion: v1
kind: Pod
metadata:
  name: dns-example
spec:
  containers:
    - name: test
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 10.96.0.10
    searches:
      - suffix1.povilasv.me
      - suffix2.povilasv.me
      - suffix3.povilasv.me
      - kube-system.svc.cluster.local
      - svc.cluster.local
      - cluster.local
    options:
      - name: ndots
        value: "5"

DNS client implementations

As mentioned earlier, applications typically use the C standard library (libc) for DNS interactions via the getaddrinfo function call. So, how the resolution works highly depends on the implementation. For instance, the musl library, used in Alpine images, resolves DNS queries in parallel rather than sequentially, which results in bursty DNS traffic but potentially faster resolution time.

Let’s look at a couple of experiments to see how different implementations behave in practice. First of all, let’s enable the log plugin in CoreDNS so we can see the queries. You can do that by editing coredns config map in the kube-system namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

Note: The log plugin was added to the top of this configuration file.

Also, check if you are running multiple replicas of CoreDNS. I suggest scaling down to a single one to rule out out-of-order logs in multiple replicas.

Experiment: querying non-existent domain

Setup

Let’s run two pods with custom DNS search path configuration – one with ubuntu image, which uses glibc and another with alpine, which uses musl library. See the following Pod YAML with custom configuration:

apiVersion: v1
kind: Pod
metadata:
  name: dns-example
spec:
  containers:
    - name: test
      image: ubuntu 
      # image: alpine
      command: ["sleep", "365d"]
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 10.96.0.10
    searches:
      - suffix1.povilasv.me
      - suffix2.povilasv.me
      - suffix3.povilasv.me
      - kube-system.svc.cluster.local
      - svc.cluster.local
      - cluster.local
    options:
      - name: ndots
        value: "5"
      - name: edns0

In this experiment, let’s query for a non-existent domain to check how each implementation handles the search path. This means the query should go through each element in the search path and then return an error at the end.

Results in Ubuntu

Now, let’s check the behavior in ubuntu Pod:

nslookup non-existent-domain
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find non-existent-domain: NXDOMAIN

real    0m0.112s
user    0m0.015s
sys     0m0.010s

nslookup reports a single error that it cannot find non-existent-domain.

Also, let’s check the CoreDNS logs:

[INFO] 10.244.0.5:47110 - 54356 "A IN non-existent-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.009731264s
[INFO] 10.244.0.5:49757 - 65481 "A IN non-existent-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.009955682s
[INFO] 10.244.0.5:51604 - 19480 "A IN non-existent-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.010034067s
[INFO] 10.244.0.5:38115 - 22442 "A IN non-existent-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000171197s
[INFO] 10.244.0.5:41636 - 56528 "A IN non-existent-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000188263s
[INFO] 10.244.0.5:38874 - 18964 "A IN non-existent-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000165339s
[INFO] 10.244.0.5:37926 - 46357 "A IN non-existent-domain. udp 37 false 512" NXDOMAIN qr,rd,ra 37 0.002296889s

Note that the ubuntu pod goes through each search path step by step and, at the last, tries the domain.

Results in Alpine Pod

Now, let’s try the same experiment with alpine Pod:

time nslookup non-existent-domain
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find non-existent-domain.cluster.local: NXDOMAIN
** server can't find non-existent-domain.cluster.local: NXDOMAIN
** server can't find non-existent-domain.kube-system.svc.cluster. local: NXDOMAIN
** server can't find non-existent-domain.kube-system.svc.cluster. local: NXDOMAIN
** server can't find non-existent-domain.svc.cluster. local: NXDOMAIN
** server can't find non-existent-domain.svc.cluster. local: NXDOMAIN
** server can't find non-existent-domain.suffix3.povilasv.me: NXDOMAIN
** server can't find non-existent-domain.suffix1.povilasv.me: NXDOMAIN
** server can't find non-existent-domain.suffix1.povilasv.me: NXDOMAIN
** server can't find non-existent-domain.suffix2.povilasv.me: NXDOMAIN
** server can't find non-existent-domain.suffix2.povilasv.me: NXDOMAIN
** server can't find non-existent-domain.suffix3.povilasv.me: NXDOMAIN

Command exited with non-zero status 1
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s

In alpine case, nslookup returns multiple errors. Also, let’s check the CoreDNS logs:

[INFO] 10.244.0.6:46089 - 22905 "AAAA IN non-existent-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000235549s
[INFO] 10.244.0.6:46089 - 63458 "A IN non-existent-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000331694s
[INFO] 10.244.0.6:46089 - 45359 "A IN non-existent-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000164723s
[INFO] 10.244.0.6:46089 - 19824 "AAAA IN non-existent-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000341596s
[INFO] 10.244.0.6:46089 - 21400 "AAAA IN non-existent-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.0003289s
[INFO] 10.244.0.6:46089 - 61499 "A IN non-existent-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000314972s
[INFO] 10.244.0.6:46089 - 17990 "AAAA IN non-existent-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001254302s
[INFO] 10.244.0.6:46089 - 38910 "A IN non-existent-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001700795s
[INFO] 10.244.0.6:46089 - 65270 "AAAA IN non-existent-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001481961s
[INFO] 10.244.0.6:46089 - 1303 "AAAA IN non-existent-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001683054s
[INFO] 10.244.0.6:46089 - 41363 "A IN non-existent-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.002119168s
[INFO] 10.244.0.6:46089 - 42990 "A IN non-existent-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.002805464s

These logs are out of order, suffix3 appears before suffix1, which indicates that the search path was searched in parallel. Additionally, there is some meaningful difference in the time it took to execute.

Experiment: Unresponsive nameserver

Setup

Let’s see what happens if we add a single unresponsive nameserver to resolv.conf. I added a 10.96.0.11 IP to the nameserver’s list to do that. Nothing is listening on this IP, so it should eventually time out.

apiVersion: v1
kind: Pod
metadata:
  name: alpine-dns-no-opts-example
spec:
  containers:
    - name: test
      image: alpine
      command: ["sleep", "365d"]
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 10.96.0.11
      - 10.96.0.10
    searches:
      - suffix1.povilasv.me
      - suffix2.povilasv.me
      - suffix3.povilasv.me
      - kube-system.svc.cluster.local
      - svc.cluster.local
      - cluster.local
    options:
      - name: ndots
        value: "5"
      - name: edns0

Results in Ubuntu Pod

First, let’s check the behavior in ubuntu Pod:

time nslookup non-existant-domain
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
Server:         10.96.0.10
Address:        10.96.0.10#53
Non-authoritative answer:
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
;; communications error to 10.96.0.11#53: timed out
*** Can't find non-existant-domain: No answer
real    1m0.166s
user    0m0.005s
sys     0m0.019s

It took more than a minute to get a response from DNS. Also, CoreDNS logs show that queries are executed sequentially:

[INFO] 10.244.0.6:53279 - 22687 "A IN non-existant-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001250128s
[INFO] 10.244.0.6:51904 - 46917 "A IN non-existant-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001001108s
[INFO] 10.244.0.6:50572 - 65164 "A IN non-existant-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.000559563s
[INFO] 10.244.0.6:45999 - 13649 "A IN non-existant-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.00022439s
[INFO] 10.244.0.6:57902 - 59328 "A IN non-existant-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000331656s
[INFO] 10.244.0.6:56451 - 59102 "A IN non-existant-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000227071s
[INFO] 10.244.0.6:51513 - 23939 "A IN non-existant-domain. udp 37 false 512" NOERROR qr,rd,ra 37 0.007888233s
[INFO] 10.244.0.6:58916 - 59345 "AAAA IN non-existant-domain. udp 37 false 512" NOERROR qr,rd,ra 37 0.006819041s

Results in Alpine Pod

Now, let’s test the same thing in alpine:

time nslookup non-existant-domain
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find non-existant-domain.cluster. local: NXDOMAIN
** server can't find non-existant-domain.svc.cluster. local: NXDOMAIN
** server can't find non-existant-domain.kube-system.svc.cluster. local: NXDOMAIN
** server can't find non-existant-domain.svc.cluster. local: NXDOMAIN
** server can't find non-existant-domain.kube-system.svc.cluster. local: NXDOMAIN
** server can't find non-existant-domain.cluster. local: NXDOMAIN
** server can't find non-existant-domain.suffix3.povilasv. me: NXDOMAIN
** server can't find non-existant-domain.suffix1.povilasv. me: NXDOMAIN
** server can't find non-existant-domain.suffix2.povilasv. me: NXDOMAIN
** server can't find non-existant-domain.suffix1.povilasv. me: NXDOMAIN
** server can't find non-existant-domain.suffix2.povilasv. me: NXDOMAIN
** server can't find non-existant-domain.suffix3.povilasv. me: NXDOMAIN

Command exited with non-zero status 1
real    0m 5.17s
user    0m 0.00s
sys     0m 0.00s

As you can see, we get multiple errors, but it takes only 5 seconds to return.

CoreDNS logs indicate parallel behavior:

[INFO] 10.244.0.5:57530 - 54255 "AAAA IN non-existant-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000208483s
[INFO] 10.244.0.5:57530 - 52406 "AAAA IN non-existant-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000291089s
[INFO] 10.244.0.5:57530 - 59127 "A IN non-existant-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000354392s
[INFO] 10.244.0.5:57530 - 20463 "A IN non-existant-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.00033021s
[INFO] 10.244.0.5:57530 - 50467 "AAAA IN non-existant-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000327972s
[INFO] 10.244.0.5:57530 - 22864 "A IN non-existant-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000336238s
[INFO] 10.244.0.5:57530 - 56061 "A IN non-existant-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.163484073s
[INFO] 10.244.0.5:57530 - 50980 "A IN non-existant-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.164862272s
[INFO] 10.244.0.5:57530 - 54016 "A IN non-existant-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.16639027s
[INFO] 10.244.0.5:57530 - 25172 "AAAA IN non-existant-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.166472807s
[INFO] 10.244.0.5:57530 - 27025 "AAAA IN non-existant-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.166581661s
[INFO] 10.244.0.5:57530 - 48079 "AAAA IN non-existant-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.166722883s

Experiment: Slow DNS provider

Setup

Lastly, let’s check what happens if the resolution is slow. I’ve built my own CoreDNS plugin, which sleeps for 3 seconds and then passes the query to other plugins.

Results in Ubuntu Pod

Let’s try this with ubuntu:

time nslookup non-existent-domain
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find non-existent-domain: NXDOMAIN

real    0m21.085s
user    0m0.009s
sys     0m0.015s

As you can see, it took us about 21 seconds to get a response. This happened because the DNS client had to do 7 requests, each taking 3 seconds. CoreDNS logs also show this:

[INFO] 10.244.0.8:44748 - 25660 "A IN non-existent-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.0134418s
[INFO] 10.244.0.8:50144 - 57801 "A IN non-existent-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.011464031s
[INFO] 10.244.0.8:60229 - 24566 "A IN non-existent-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.014714347s
[INFO] 10.244.0.8:37521 - 60259 "A IN non-existent-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000294787s
[INFO] 10.244.0.8:53416 - 46788 "A IN non-existent-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000300435s
[INFO] 10.244.0.8:60904 - 51233 "A IN non-existent-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000184105s
[INFO] 10.244.0.8:52037 - 18022 "A IN non-existent-domain. udp 37 false 512" NXDOMAIN qr,rd,ra 37 0.007187617s

Results in Alpine Pod

Now, let’s test the same thing in alpine:

/ # time nslookup non-existent-domain
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find non-existent-domain.kube-system.svc.cluster. local: NXDOMAIN
** server can't find non-existent-domain.cluster. local: NXDOMAIN
** server can't find non-existent-domain.svc.cluster. local: NXDOMAIN
** server can't find non-existent-domain.cluster. local: NXDOMAIN
** server can't find non-existent-domain.svc.cluster. local: NXDOMAIN
** server can't find non-existent-domain.kube-system.svc.cluster. local: NXDOMAIN
** server can't find non-existent-domain.suffix1.povilasv. me: NXDOMAIN
** server can't find non-existent-domain.suffix3.povilasv. me: NXDOMAIN
** server can't find non-existent-domain.suffix1.povilasv. me: NXDOMAIN
** server can't find non-existent-domain.suffix2.povilasv. me: NXDOMAIN
** server can't find non-existent-domain.suffix2.povilasv. me: NXDOMAIN
** server can't find non-existent-domain.suffix3.povilasv. me: NXDOMAIN

Command exited with non-zero status 1
real    0m 3.00s
user    0m 0.00s
sys     0m 0.00s

Note that alpine only took 3 seconds to complete the request as everything was evaluated in parallel. Also, the following COREDNS logs show that Alpine nslookup sent way more queries to CoreDNS.

[INFO] 10.244.0.7:47024 - 45874 "AAAA IN non-existent-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000323515s
[INFO] 10.244.0.7:47024 - 19996 "A IN non-existent-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000180658s
[INFO] 10.244.0.7:47024 - 47909 "AAAA IN non-existent-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000442295s
[INFO] 10.244.0.7:47024 - 49715 "AAAA IN non-existent-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.00050683s
[INFO] 10.244.0.7:47024 - 17452 "A IN non-existent-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000419432s
[INFO] 10.244.0.7:47024 - 54020 "A IN non-existent-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000374228s
[INFO] 10.244.0.7:47024 - 46061 "A IN non-existent-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.00140339s
[INFO] 10.244.0.7:47024 - 51134 "A IN non-existent-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001460635s
[INFO] 10.244.0.7:47024 - 22357 "AAAA IN non-existent-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001842863s
[INFO] 10.244.0.7:47024 - 24217 "AAAA IN non-existent-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001795028s
[INFO] 10.244.0.7:47024 - 49078 "A IN non-existent-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001771025s
[INFO] 10.244.0.7:47024 - 43665 "AAAA IN non-existent-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,rd,ra 145 0.001774173s
[INFO] 10.244.0.7:47024 - 46061 "A IN non-existent-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,aa,rd,ra 145 0.000208325s
[INFO] 10.244.0.7:47024 - 17452 "A IN non-existent-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000166721s
[INFO] 10.244.0.7:47024 - 22357 "AAAA IN non-existent-domain.suffix1.povilasv.me. udp 57 false 512" NXDOMAIN qr,aa,rd,ra 145 0.000191355s
[INFO] plugin/example: example
[INFO] 10.244.0.7:47024 - 49078 "A IN non-existent-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,aa,rd,ra 145 0.000170383s
[INFO] 10.244.0.7:47024 - 19996 "A IN non-existent-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000205738s
[INFO] 10.244.0.7:47024 - 24217 "AAAA IN non-existent-domain.suffix2.povilasv.me. udp 57 false 512" NXDOMAIN qr,aa,rd,ra 145 0.000595005s
[INFO] 10.244.0.7:47024 - 49715 "AAAA IN non-existent-domain.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000490129s
[INFO] 10.244.0.7:47024 - 54020 "A IN non-existent-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.00064143s
[INFO] 10.244.0.7:47024 - 43665 "AAAA IN non-existent-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,aa,rd,ra 145 0.000565785s
[INFO] 10.244.0.7:47024 - 45874 "AAAA IN non-existent-domain.kube-system.svc.cluster.local. udp 67 false 512" NXDOMAIN qr,aa,rd 160 0.000687267s
[INFO] 10.244.0.7:47024 - 51134 "A IN non-existent-domain.suffix3.povilasv.me. udp 57 false 512" NXDOMAIN qr,aa,rd,ra 145 0.000865276s
[INFO] 10.244.0.7:47024 - 47909 "AAAA IN non-existent-domain.svc.cluster.local. udp 55 false 512" NXDOMAIN qr,aa,rd 148 0.000825555s

Other considerations

Actually, the application is not required to call the C standard library. For example, in Go, you can compile a program with a netgo build tag that uses a Go-based DNS resolution implementation instead of calling it out to libc. It has advantages over calling out to libc, such that a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread. So, the DNS resolution highly depends on your application runtime configuration and could behave differently than musl or standard glibc.

go build -tags netgo main.go

Example: building a Go application that uses Go-based DNS resolution.

Summary

This blog post explores the crucial role of DNS in Kubernetes, focusing on service discovery and CoreDNS as the default DNS provider. We delve into DNS resolution mechanics Kubernetes DNS policies like ClusterFirst, Default, and None, and their impact on DNS queries within Pods. Through detailed experiments, we demonstrate how DNS client implementations and configurations affect service discovery performance, providing insights that help Kubernetes users optimize applications and troubleshoot DNS-related issues effectively. Choosing the correct DNS settings has a real-world impact on application performance.

References:

Sign up and never miss an article

Povilas

About the Author

I'm Povilas Versockas, a software engineer, blogger, Certified Kubernetes Administrator, CNCF Ambassador, and a computer geek.