CODEWITHSUNDEEP
dnskubernetesskydns
user4889345
user4889345

Reputation:

Unable to setup service DNS in Kubernetes cluster

Kubernetes version --> 1.5.2

I am setting up DNS for Kubernetes services for the first time and I came across SkyDNS. So following documentation, my skydns-svc.yaml file is :

apiVersion: v1
kind: Service
spec:
  clusterIP: 10.100.0.100
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  selector:
    k8s-app: kube-dns
  sessionAffinity: None
  type: ClusterIP

And my skydns-rc.yaml file is :

apiVersion: v1
kind: ReplicationController
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v18
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kube-dns
        kubernetes.io/cluster-service: "true"
        version: v18
    spec:
      containers:
      - args:
        - --domain=kube.local
        - --dns-port=10053
        image: gcr.io/google_containers/kubedns-amd64:1.6
        imagePullPolicy: IfNotPresent
        name: kubedns
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
      - args:
        - --cache-size=1000
        - --no-resolv
        - --server=127.0.0.1#10053
        image: gcr.io/google_containers/kube-dnsmasq-amd64:1.3
        imagePullPolicy: IfNotPresent
        name: dnsmasq
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
      - args:
        - -cmd=nslookup kubernetes.default.svc.kube.local 127.0.0.1 >/dev/null &&
          nslookup kubernetes.default.svc.kube.local 127.0.0.1:10053 >/dev/null
        - -port=8080
        - -quiet
        image: gcr.io/google_containers/exechealthz-amd64:1.0
        imagePullPolicy: IfNotPresent
        name: healthz
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

Also on my minions, I updated the /etc/systemd/system/multi-user.target.wants/kubelet.service file and added the following under the ExecStart section :

ExecStart=/usr/bin/kubelet \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBELET_API_SERVER \
        $KUBELET_ADDRESS \
        $KUBELET_PORT \
        $KUBELET_HOSTNAME \
        $KUBE_ALLOW_PRIV \
        $KUBELET_POD_INFRA_CONTAINER \
        $KUBELET_ARGS \
            --cluster-dns=10.100.0.100 \
            --cluster-domain=kubernetes \

Having done all of this and having successfully brought up the rc & svc :

[root@kubernetes-master DNS]# kubectl get po | grep dns
kube-dns-v18-hl8z6                                                3/3       Running             0          6s
[root@kubernetes-master DNS]# kubectl get svc | grep dns
kube-dns                            10.100.0.100     <none>        53/UDP,53/TCP                                                    20m

This is all that I got from a config standpoint. Now in order to test my setup, I downloaded busybox and tested a nslookup

[root@kubernetes-master DNS]# kubectl get svc | grep kubernetes
kubernetes                          10.100.0.1       <none>        443/TCP 

[root@kubernetes-master DNS]# kubectl exec busybox -- nslookup kubernetes
nslookup: can't resolve 'kubernetes'
Server:    10.100.0.100
Address 1: 10.100.0.100

Is there something that I have missed ?

EDIT ::

Going through the logs, I see something that might explain why this is not working :

kubectl logs $(kubectl get pods -l k8s-app=kube-dns -o name) -c kubedns
.
.
.
E1220 17:44:48.403976       1 reflector.go:216] pkg/dns/dns.go:154: Failed to list *api.Endpoints: Get https://10.100.0.1:443/api/v1/endpoints?resourceVersion=0: x509: failed to load system roots and no roots provided
E1220 17:44:48.487169       1 reflector.go:216] pkg/dns/dns.go:155: Failed to list *api.Service: Get https://10.100.0.1:443/api/v1/services?resourceVersion=0: x509: failed to load system roots and no roots provided
I1220 17:44:48.487716       1 dns.go:172] Ignoring error while waiting for service default/kubernetes: Get https://10.100.0.1:443/api/v1/namespaces/default/services/kubernetes: x509: failed to load system roots and no roots provided. Sleeping 1s before retrying.
E1220 17:44:49.410311       1 reflector.go:216] pkg/dns/dns.go:154: Failed to list *api.Endpoints: Get https://10.100.0.1:443/api/v1/endpoints?resourceVersion=0: x509: failed to load system roots and no roots provided
I1220 17:44:49.492338       1 dns.go:172] Ignoring error while waiting for service default/kubernetes: Get https://10.100.0.1:443/api/v1/namespaces/default/services/kubernetes: x509: failed to load system roots and no roots provided. Sleeping 1s before retrying.
E1220 17:44:49.493429       1 reflector.go:216] pkg/dns/dns.go:155: Failed to list *api.Service: Get https://10.100.0.1:443/api/v1/services?resourceVersion=0: x509: failed to load system roots and no roots provided
.
.
.

Looks like kubedns is unable to authorize against K8S master node. I even tried to do a manual call :

curl -k https://10.100.0.1:443/api/v1/endpoints?resourceVersion=0
Unauthorized

Upvotes: 0

Views: 276

Answers (1)

Vikram Hosakote
Vikram Hosakote

Reputation: 3664

Looks like the kube-dns pod is not able to authenticate with the kubernetes api server. I don't see any secret and serviceaccount in the YAML file for the kube-dns pod.

I suggest doing the following:

  1. Create a k8s secret using kubectl create secret for the kube-dns pod with the right certificate file ca.crt and token:

    $ kubectl get secrets -n=kube-system | grep dns
kube-dns-token-66tfx     kubernetes.io/service-account-token   3      1d

  2. Create a k8s serviceaccount using kubectl create serviceaccount for the kube-dns pod:

    $ kubectl get serviceaccounts -n=kube-system | grep dns
kube-dns               1         1d`

  3. Mount the secret at /var/run/secrets/kubernetes.io/serviceaccount inside the kube-dns container in the YAML file:

    ...
kind: Pod
...
spec:
  ...
  containers:
    ...
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-dns-token-66tfx
      readOnly: true
  ...
  volumes:
  - name: kube-dns-token-66tfx
    secret:
      defaultMode: 420
      secretName: kube-dns-token-66tfx

Here are the links about creating serviceaccounts for pods:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

https://kubernetes.io/docs/admin/service-accounts-admin/

Upvotes: 1

Related Questions