Trying to pull/run docker images from docker hub on Minikube fails

Question

I am very new to Kuberetes and I have done some work with docker previously. I am trying to accomplish following:

1. Spin up Minikube
2. Use Kube-ctl to spin up a docker image from docker hub.

I started minikube and things look like they are up and running. Then I pass following command

kubectl run nginx --image=nginx (Please note I do not have this image anywhere on my machine and I am expecting k8 to fetch it for me)

Now, when I do that, it spins up the pod but the status is ImagePullBackOff. So I ran kubectl describe pod command on it and the results look like following:

Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  8m               default-scheduler  Successfully assigned default/ngix-67c6755c86-qm5mv to minikube
  Warning  Failed     8m               kubelet, minikube  Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:52133->192.168.64.1:53: read: connection refused
  Normal   Pulling    8m (x2 over 8m)  kubelet, minikube  Pulling image "nginx"
  Warning  Failed     8m (x2 over 8m)  kubelet, minikube  Error: ErrImagePull
  Warning  Failed     8m               kubelet, minikube  Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:40073->192.168.64.1:53: read: connection refused
  Normal   BackOff    8m (x3 over 8m)  kubelet, minikube  Back-off pulling image "nginx"
  Warning  Failed     8m (x3 over 8m)  kubelet, minikube  Error: ImagePullBackOff

Then I searched around to see if anyone has faced similar issues and it turned out that some people have and they did resolve it by restarting minikube using some more flags which look like below:

minikube start --vm-driver="xhyve" --insecure-registry="$REG_IP":80

when I do nslookup inside Minikube, it does resolve with following information:

Server:         10.12.192.22
Address:        10.12.192.22#53

Non-authoritative answer:
hub.docker.com  canonical name = elb-default.us-east-1.aws.dckr.io.
elb-default.us-east-1.aws.dckr.io       canonical name = us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.
Name:   us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 52.205.36.130
Name:   us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 3.217.62.246
Name:   us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com
Address: 35.169.212.184

still no luck. Is there anything that I am doing wrong here?

Answer 1

You have:

1. minukube running with default settings.
2. docker building your images
3. (*) configured minikube to point to your docker images local repo

And now minikube can't pull images from public "container" registries, like docker hub.

stop and start minikube, then point it back to your local docker images repo. The commands to do this (and (*) this):

minikube stop
minikube start
minikube -p minikube docker-env
eval $(minikube -p minikube docker-env)

Since running the above I was able to pull nginx, alpine and frens from hub.docker.come just by setting image: alpine in the yaml spec.

The issue was just a short drop in my network connectivity. So if you have no dns/vpn/xhyve complications and it just stops, the fix is easy enough.

Answer 2

I got this problem with my local minikube setup and I wasn't able to pull any images I added to a simple deployment manifest.

$ kubectl get pods
NAME              READY   STATUS             RESTARTS   AGE
test1             0/1     ImagePullBackOff   0          68s

Tried to execute the below test:

apiVersion: v1
kind: Pod
metadata:
  name: test1
  labels:
    site: blog
spec:
  containers:
    - name: web
      image: nginx:latest

It was possible or fixed only after restarting the minikube. Maybe the dnsmasq was really the cause in this case.

Answer 3

In my case it was caused by running dnsmasq, a dns server, on my Mac using Homebrew, which caused the DNS requests to fail inside minikube. After stopping dnsmasq, everything worked.

Answer 4

There error message suggests that the Docker daemon running in the minikube VM can't resolve the registry-1.docker.io hostname because the DNS nameserver it's configured to use for DNS resolution (192.168.64.1:53) is refusing connection. It's strange to me that the Docker deamon is trying to resolve registry-1.docker.io via a nameserver at 192.168.64.1 but when you nslookup on the VM it's using a nameserver at 10.12.192.22. I did an Internet search for "minkube Get registry-1.docker.io/v2: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53" and found an issue where someone made this comment, seems identical to your problem, and seems specific to xhyve.

In that comment the person says:

This issue does look like an xhyve issue not seen with virtualbox.

and

Switching to virtualbox fixed this issue for me.

I stopped minikube, deleted it, started it without --vm-driver=xhyve (minikube uses virtualbox driver by default), and then docker build -t hello-node:v1 . worked fine without errors