Reputation: 11337

What is the meaning of ImagePullBackOff status on a Kubernetes pod?

I'm trying to run my first kubernetes pod locally. I've run the following command (from here):

export ARCH=amd64
docker run -d \
    --volume=/:/rootfs:ro \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:rw \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
    --volume=/var/run:/var/run:rw \
    --net=host \
    --pid=host \
    --privileged \
    gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \
    /hyperkube kubelet \
        --containerized \
        --hostname-override=127.0.0.1 \
        --api-servers=http://localhost:8080 \
        --config=/etc/kubernetes/manifests \
        --cluster-dns=10.0.0.10 \
        --cluster-domain=cluster.local \
        --allow-privileged --v=2

Then, I've trying to run the following:

kubectl create -f ./run-aii.yaml

run-aii.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: aii
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: aii
    spec:
      containers:
      - name: aii
        image: aii
        ports:
        - containerPort: 5144
        env:
        - name: KAFKA_IP
          value: kafka
        volumeMounts:
        - mountPath: /root/script
          name: scripts-data
          readOnly: true
        - mountPath: /home/aii/core
          name: core-aii
          readOnly: true
        - mountPath: /home/aii/genome
          name: genome-aii
          readOnly: true
        - mountPath: /home/aii/main
          name: main-aii
          readOnly: true
      - name: kafka
        image: kafkazoo
        volumeMounts:
        - mountPath: /root/script
          name: scripts-data
          readOnly: true
        - mountPath: /root/config
          name: config-data
          readOnly: true
      - name: ws
        image: ws
        ports:
        - containerPort: 3000
      volumes:
      - name: scripts-data
        hostPath:
          path: /home/aii/general/infra/script
      - name: config-data
        hostPath:
          path: /home/aii/general/infra/config
      - name: core-aii
        hostPath: 
          path: /home/aii/general/core
      - name: genome-aii
        hostPath: 
          path: /home/aii/general/genome
      - name: main-aii
        hostPath: 
          path: /home/aii/general/main

Now, when I run: kubectl get pods I'm getting:

NAME                    READY     STATUS             RESTARTS   AGE
aii-806125049-18ocr     0/3       ImagePullBackOff   0          52m
aii-806125049-6oi8o     0/3       ImagePullBackOff   0          52m
aii-pod                 0/3       ImagePullBackOff   0          23h
k8s-etcd-127.0.0.1      1/1       Running            0          2d
k8s-master-127.0.0.1    4/4       Running            0          2d
k8s-proxy-127.0.0.1     1/1       Running            0          2d
nginx-198147104-9kajo   1/1       Running            0          2d

BTW: docker images return:

REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
ws                                         latest              fa7c5f6ef83a        7 days ago          706.8 MB
kafkazoo                                   latest              84c687b0bd74        9 days ago          697.7 MB
aii                                        latest              bd12c4acbbaf        9 days ago          1.421 GB
node                                       4.4                 1a93433cee73        11 days ago         647 MB
gcr.io/google_containers/hyperkube-amd64   v1.2.4              3c4f38def75b        11 days ago         316.7 MB
nginx                                      latest              3edcc5de5a79        2 weeks ago         182.7 MB
docker_kafka                               latest              e1d954a6a827        5 weeks ago         697.7 MB
spotify/kafka                              latest              30d3cef1fe8e        12 weeks ago        421.6 MB
wurstmeister/zookeeper                     latest              dc00f1198a44        3 months ago        468.7 MB
centos                                     latest              61b442687d68        4 months ago        196.6 MB
centos                                     centos7.2.1511      38ea04e19303        5 months ago        194.6 MB
gcr.io/google_containers/etcd              2.2.1               a6cd91debed1        6 months ago        28.19 MB
gcr.io/google_containers/pause             2.0                 2b58359142b0        7 months ago        350.2 kB
sequenceiq/hadoop-docker                   latest              5c3cc170c6bc        10 months ago       1.766 GB

why do I get the ImagePullBackOff ??

Upvotes: 129

Answers (17)

Muhammed Moussa

Reputation: 5195

for me I was logged in to a docker private registry and tried to pull the public image, so tried docker logout solved the issue for me.

so you can also validate the images you are using if public or private, then try docker login or docker logout upon that.

Upvotes: 2

Rohit Parte

Reputation: 4036

To handle this error, Just have to create Kubernetes secrets and use it in manifest.yaml file

If it is private repository then it is mandatory to use user secrets

To generate secrets -

kubectl create secret docker-registry docker-secrets --docker-server=https://index.docker.io/v1/ --docker-username=ExamplaName --docker-password=ExamplePassword [email protected]

for --docker-server, use https://index.docker.io/v1/

apiVersion: v1
kind: Pod
metadata:
   name: test-pod
spec:
   containers:
   - name: test
     image: ExampleUsername/test:tagname
     ports:
     - containerPort: 3015
   imagePullSecrets:
   - name: docker-secrets

Upvotes: 1

Raghunath Babalsure

Reputation: 115

ImagepullBackoff mesns you have not passed secret in your yaml or secret is wrong and might be you image name is wrong.

If you pulling image from private registry you have to provide image pull secret then it will able to pull image.

you also need to creat secrete before you deploy the pod. you can use below command to create secrete.

 kubectl create secret docker-registry regcred --docker-server=artifacts.exmple.int --docker-username=<username> --docker-password=<password> -n <namespace>

you can pass secret in yaml like below.

 imagePullSecrets:
 - name: regcred

Upvotes: 2

Raghunath Babalsure

Reputation: 115

I had face same issue.

imagePullBackOff means it is not able to pull docker image from registry or smoking issue with your registry.

the solution would be as below.

1.  Check you image registry name.
2.  check image pull secrets.
3.  check image is present with same tag or name.
4.  check you registry is working.

Upvotes: 1

ARKhan

Reputation: 1975

For my case, Kubernetes was not able to communicate to my private registry running on localhost:5000 after update to MacOS Monterey. It was running fine previously. The reason was Apple Airplay now listen to port 5000. In order to resolve this issue, I disabled Apple Airplay receiver.

Go To System preference > Sharing > Disable checkbox for Airplay receiver.

Source Link: https://developer.apple.com/forums/thread/682332

Upvotes: 0

ryanwebjackson

Reputation: 1046

Debugging step:
kubectl get pod [name] -o yaml

Run this command to get the YAML configuration of the pod (Get YAML for deployed Kubernetes services?). In my case, it was under this section:

state:
waiting:
message: 'rpc error: code = Unknown desc = Error response from daemon: Get https://repository:9999/v2/abc/location/image/manifests/tag: unauthorized: BAD_CREDENTIAL'
reason: ErrImagePull

Upvotes: 5

mayur murkya

Reputation: 149

I had the same issue.

[mayur@mayur_cloudtest ~]$ kubectl get pods
NAME                     READY   STATUS              RESTARTS   AGE
nginx-598b589c46-zcr5d   0/1     ImagePullBackOff    0          6m21s

Later I found that the docker on which the pod is created is using a private registry for images and Nginx was not present in it.

I have changed the docker registry to default and reloaded the daemon. Post that issue got resolved.

[mayur@mayur_cloudtest ~]$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-598b589c46-7cbjf   1/1     Running   0          33s
[mayur@mayur_cloudtest ~]$
[mayur@mayur_cloudtest ~]$
[mayur@mayur_cloudtest ~]$ kubectl exec -it nginx-598b589c46-7cbjf -- /bin/bash
root@nginx-598b589c46-7cbjf:/# ls
bin   dev                  docker-entrypoint.sh  home  lib64  mnt  proc  run   srv  tmp  var
boot  docker-entrypoint.d  etc                   lib   media  opt  root  sbin  sys  usr
root@nginx-598b589c46-7cbjf:/#

Upvotes: 0

vagdevi

Reputation: 51

My issue got resolved upon adding the appropriate tag to the image I wanted to pull from the DockerHub.

Previously:

containers:
      - name: nginx
        image: alex/my-app-image

Corrected Version:

containers:
      - name: nginx
        image: alex/my-app-image:1.1

The image has only one version, which was 1.1. Since I skipped that initially, it has thrown an error.

After correctly mentioning the version, it worked fine!!

Upvotes: 3

Jacek Laskowski

Reputation: 74669

Despite all the other great answers none helped me until I found a comment that pointed out this Updating images:

The default pull policy is IfNotPresent which causes the kubelet to skip pulling an image if it already exists.

That's exactly what I wanted, but didn't seem to work.

Reading further said the following:

If you would like to always force a pull, you can do one of the following:

omit the imagePullPolicy and use :latest as the tag for the image to use.

When I replaced latest with a version (that I had pushed to minikube's Docker daemon), it worked fine.

$ kubectl create deployment presto-coordinator \
    --image=warsaw-data-meetup/presto-coordinator:beta0
deployment.apps/presto-coordinator created

$ kubectl get deployments
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
presto-coordinator   1/1     1            1           3s

Find the pod of the deployment (using kubectl get pods) and use kubectl describe pod to find out more on the pod.

Upvotes: 3

Sachin Mishra

Reputation: 1183

The issue arises when the image is not present on the cluster and k8s engine is going to pull the respective registry. k8s Engine enables 3 types of ImagePullPolicy mentioned :

Always : It always pull the image in container irrespective of changes in the image
Never : It will never pull the new image on the container
IfNotPresent : It will pull the new image in cluster if the image is not present.

Best Practices : It is always recommended to tag the new image in both docker file as well as k8s deployment file. So That it can pull the new image in container.

Upvotes: 11

Nithya

Reputation: 782

I too had this problem, when I checked I image that I was pulling from a private registry was removed If we describe pod it will show pulling event and the image it's trying to pull

kubectl describe pod <POD_NAME>

Events:
 Type     Reason   Age                  From              Message
 ----     ------   ----                 ----              -------
 Normal   Pulling  18h (x35 over 20h)   kubelet, gsk-kub  Pulling image "registeryName:tag"
 Normal   BackOff  11m (x822 over 20h)  kubelet, gsk-kub  Back-off pulling image "registeryName:tag"
 Warning  Failed   91s (x858 over 20h)  kubelet, gsk-kub  Error: ImagePullBackOff

Upvotes: 11

Naveen Sadasivam

Reputation: 29

I had this error when I tried to create a replicationcontroller. The issue was, I wrongly spelt the nginx image name in template definition.

Note: This error occurs when kubernetes is unable to pull the specified image from the repository.

Upvotes: 0

Margach Chris

Reputation: 1739

One issue that may cause an ImagePullBackOff especially if you are pulling from a private registry is if the pod is not configured with the imagePullSecret of the private registry.

An authentication error may cause an imagePullBackOff.

Upvotes: 27

Richard

Reputation: 4840

You can specify also imagePullPolicy: Never in the container's spec:

containers:
- name: nginx
  imagePullPolicy: Never
  image: custom-nginx
  ports:
  - containerPort: 80

Upvotes: 15

Ranvir

Reputation: 2142

I had similar problem when using minikube over hyperv with 2048GB memory. I found that in HyperV manager the Memory Demand was higher than allocated.

So I stopped minikube and assigned somewhere between 4096-6144GB. It worked fine after that, all pods running!

I don't know if this can nail down the issue in every case. But just have a look at the memory and disk allocated to the minikube.

Upvotes: 1

user9200729

Reputation: 149

I had the same problem what caused it was that I already had created a pod from the docker image via the .yml file, however I mistyped the name, i.e test-app:1.0.1 when I needed test-app:1.0.2 in my .yml file. So I did kubectl delete pods --all to remove the faulty pod then redid the kubectl create -f name_of_file.yml which solved my problem.

Upvotes: 14

Pixel Elephant

Reputation: 21383

By default Kubernetes looks in the public Docker registry to find images. If your image doesn't exist there it won't be able to pull it.

You can run a local Kubernetes registry with the registry cluster addon.

Then tag your images with localhost:5000:

docker tag aii localhost:5000/dev/aii

Push the image to the Kubernetes registry:

docker push localhost:5000/dev/aii

And change run-aii.yaml to use the localhost:5000/dev/aii image instead of aii. Now Kubernetes should be able to pull the image.

Alternatively, you can run a private Docker registry through one of the providers that offers this (AWS ECR, GCR, etc.), but if this is for local development it will be quicker and easier to get setup with a local Kubernetes Docker registry.

Upvotes: 56

What is the meaning of ImagePullBackOff status on a Kubernetes pod?

Answers (17)

Related Questions