Reputation: 19695
CoreDNS Not resolving service url outside namespace with K8S / Minikube
I have a local cluster with minikube 1.6.2 running.
All my pods are OK, I checked the logs individually, but I have 2 db, influx and postgres, are not accesible anymore from any url outside namespace.
I logged into both pods, and I can confirm that each db is OK, has data, and I can connect manually with my user / pass.
Let's take the case of influx.
kubectl exec -it -n influx blockchain-influxdb-local-fb745b98c-vbghp -- influx -username='myuser' -password="mypass" -database="mydb" -precision=rfc3339 -execute "show measurements"
gives me 4 measurements, so no pb.
but when I try to connect influx from the same namespace with his local dns, I get a timeout.
➜ ~ kubectl get svc -n influx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
blockchain-influxdb-local ClusterIP 10.96.175.62 <none> 8086/TCP 19m
➜ ~ kubectl get deployments -n influx
NAME READY UP-TO-DATE AVAILABLE AGE
blockchain-influxdb-local 1/1 1 1 20m
➜ ~ kubectl get po -n influx
NAME READY STATUS RESTARTS AGE
blockchain-influxdb-local-fb745b98c-vbghp 1/1 Running 0 21m
measures-api-local-8667bb496f-4wp8d 1/1 Running 0 21m
Case where it works:
From a pod inside the same namespace:
curl --verbose -G 'http://blockchain-influxdb-local:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
From a pod in another namespace (same namespace), with pod IP
curl --verbose -G '172.17.0.5:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
From a pod in another namespace (same namespace), with service IP
curl --verbose -G '10.96.175.62:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
But when I use local dns from outside namespace, it won't work, I get a timeout from CURL:
curl --verbose -G 'blockchain-influxdb-local.influx.svc.cluster.local:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
I followed those debug step to ensure DNS is working, and had no problem, everything works.
https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
Inside same pod, when I ping this url, I get:
root@metadata-api-local-8b4b7846b-zllb8:/go/src/gitlab.com/company/metadata_api# ping blockchain-influxdb-local.influx.svc.cluster.local
PING nc-ass-vip.sdv.fr (212.95.74.75) 56(84) bytes of data.
--- nc-ass-vip.sdv.fr ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 47ms
I don't know why is it making a reference to nc-ass-vip.sdv.fr
I also tried to remove local cluster and redeploy it, also tried to update minikube to latest version (1.8.2), nothing worked.
I don't know what else to do...
Has anyone an idea ? I was working well for monthes, don't really know what happened. :(
In response to @Arghya Sadhu, I post the file /etc/resolv.conf from the Influx pod:
nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:5
kubectl edit cm coredns -n kube-system
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2020-03-19T10:59:28Z"
name: coredns
namespace: kube-system
resourceVersion: "176"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 0797c1a9-e9db-4b4c-bc8d-4c7ecca24968
EDIT:
kubectl exec -ti dnsutils -- nslookup blockchain-influxdb-local.influx.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr canonical name = nc-ass-vip.sdv.fr.
Name: nc-ass-vip.sdv.fr
Address: 212.95.74.75
Upvotes: 1
Views: 2150
Answers (1)
Reputation: 5950
After digging into a few possibilities we came across the output for the following commands:
$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx
$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local
$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.
Output for these commands here (adding to the end of this answer for future reference in case of link doesn't work).
Reviewing this output we can see that no matter what numericable.fr is always giving positive answer to dns queries.
To avoid this situation you can change ndots entry to 1 or even 0 in your pods.
nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:0
From man pages we have:
ndots:n Sets a threshold for the number of dots which must appear in a name given to res_query(3) (see resolver(3)) before an initial absolute query will be made. The default for n is 1, meaning that if there are any dots in a name, the name will be tried first as an absolute name before any search list elements are appended to it. The value for this option is silently capped to 15.
A more effective and long term solution is to add this entry in the pod/statefulset/deployment manifest as in this example:
apiVersion: v1
kind: Pod
metadata:
namespace: default
name: dns-example
spec:
containers:
- name: test
image: nginx
dnsConfig:
options:
- name: ndots
value: "0"
Output from commands referenced for future reference:
➜ ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.default.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1584628757
refresh = 7200
retry = 1800
expire = 86400
minimum = 30
ttl = 10
ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.default.svc.cluster.local: NXDOMAIN
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
ANSWERS:
-> blockchain-influxdb-local.influx.svc.cluster.local
internet address = 10.96.72.6
ttl = 10
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Name: blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6
pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜ ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1584628757
refresh = 7200
retry = 1800
expire = 86400
minimum = 30
ttl = 30
ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local: NXDOMAIN
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1584628757
refresh = 7200
retry = 1800
expire = 86400
minimum = 30
ttl = 30
ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local: NXDOMAIN
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1584628757
refresh = 7200
retry = 1800
expire = 86400
minimum = 30
ttl = 30
ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.cluster.local: NXDOMAIN
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr, type = A, class = IN
ANSWERS:
-> blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr
canonical name = nc-ass-vip.sdv.fr.
ttl = 30
-> nc-ass-vip.sdv.fr
internet address = 212.95.74.75
ttl = 30
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr canonical name = nc-ass-vip.sdv.fr.
Name: nc-ass-vip.sdv.fr
Address: 212.95.74.75
pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜ ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
ANSWERS:
-> blockchain-influxdb-local.influx.svc.cluster.local
internet address = 10.96.72.6
ttl = 30
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Name: blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6
pod "dnsutils" deleted
Upvotes: 1
Related Questions
- Resolving external domains from within pods does not work
- Can't resolve dns in kubernetes
- Can't resolve dns from inside k8s pod
- Kubernetes DNS not resolving
- External dns lookup fails in pods
- Minikube's CoreDNS fails to resolve names from JVM
- Why coredns in Kubernetes can't resolve Pod DNS Records
- ExternalName pointing to service in other namespace doesn't resolve unless given full FQDN
- coredns do not resolve service name correctly
- Pods do not resolve the domain names of a service through ingress