Unable to use images from local docker inside k8s node ( pre pull image )

Question

We have built k8s using kubeadm and setup 3 nodes machines (which we have root access to)

We want to use the feature of k8s that they call according to their docs "pre-pull images" which is that pods that will be created on nodes - will use the local docker images vs the public internet.

Problem is it's not working.

Running the below command on all nodes - will have the same result

$ sudo docker images
REPOSITORY                           TAG       IMAGE ID       CREATED          SIZE
busybox                              latest    65ad0d468eb1   12 months ago    4.26MB
mybusybox                            latest    65ad0d468eb1   12 months ago    4.26MB

I tagged the busybox -> mybusybox just to make sure that the pod isn't taking the busybox image from Docker Hub.

So i'm trying to run k8s with the following:

kubectl run hello-world-0 -ti --image=mybusybox --image-pull-policy=Never --restart=Never

When running kubectl describe hello-world-0 I see that it got assigned to a node that i'm 100% sure the image exists on ( per output above )

But I'm receiving the following error

Events:
  Type     Reason             Age                   From               Message
  ----     ------             ----                  ----               -------
  Normal   Scheduled          2m33s                 default-scheduler  Successfully assigned ns-test/hello-world-0 to ip-172-x-x-x.europe.compute.internal
  Warning  Failed             32s (x12 over 2m32s)  kubelet            Error: ErrImageNeverPull
  Warning  ErrImageNeverPull  19s (x13 over 2m32s)  kubelet            Container image "mybusybox" is not present with pull policy of Never

---

I will just note, that also from the node itself when I try running:

node1$ sudo docker pull mybusybox
Using default tag: latest
Error response from daemon: pull access denied for mybusybox, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

but I'm not really sure if this error is related to the error that k8s receive in the pods ? We never configured any credentials, we just installed Docker so I'm not sure that the login is the issue ...

Appreciate any help here ...

Answer 1

The issue was a DNS issue, specifically on CentOS 7+

K8S has its own internal DNS service that creates domains automatically so that pods can talk to services etc w/o specifying a static IP ( which might change when you stop/start services )

I'm talking about the domains that are constructed by:

<service-name>.<namespace>.svc.cluster.local:<service-port>

That service (created by coredns or kube-dns)

• created domains in network: 192.x.x.x
• But pods were created in network: 10.x.x.x

Github bug was reported here: Fresh deploy with CoreDNS not resolving any dns lookup #1056 https://github.com/kubernetes/kubeadm/issues/1056

What solved it, was to upgrade the cluster component to a specific version. meaning, delete the entire cluster, upgrade/reinstall and recreate it.

Answer 2

Kubelet will try to pull the image from public docker repo if the repository name is not specified. More details of this is mentioned in k8s documentation.

Please push the images to private repo which needs to used. Pull the images into nodes manually. Post that image pull policy "Never" can be used.

Below is sample kubectl run hello-world-0 --image=private-repo-name/mybusybox --image-pull-policy=Never --restart=Never

Answer 3

This issue happened because the image was not pushed to the repository.

Make sure your image is tagged as mybusybox:latest and imagePullPolicy is set to Always.

Execute this command: docker push mybusybox:latest

Hopefully, this will help you.