CODEWITHSUNDEEP
kuberneteskubernetes-pod
Chetan
Chetan

Reputation: 41

kubelet fails the running pods even image is present at worker

Could you guys please help with a confusion of image pull in k8s?

  1. imagePullPolicy: Always => is set in deployment
  2. image with tag is deployed => nexus.example.com/abc-image:
  3. deployment is running with >1 replicas, lets assume 4 K8s architecture has 10 worker nodes in cluster which can run above deployment, k8s version: 1.17

The deployment/pods are running perfectly fine. What if the image is deleted from the nexus storage? Would it impact the running pods on worker? Would the running pods be failed because of ImagePullBackOff as nexus doesn't have that manifest anymore? I am asking this because logs says so. I always thought running pods wouldn't fail until local image on worker is deleted and pods/container gets restarted.

Apr 14 18:25:34 npl9app17 kubelet[207293]: E0414 18:25:34.142584  207293 remote_image.go:113] PullImage "nexus.example.com/abc-web-service-prod:5.15_130" from image service failed: rpc error: code = Unknown desc = Error response from daemon: manifest for nexus.example.com/abc-web-service-prod:5.15_130 not found: manifest unknown: manifest unknown
Apr 14 18:25:34 npl9app17 kubelet[207293]: E0414 18:25:34.142698  207293 kuberuntime_manager.go:803] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: manifest for nexus.example.com/abc-web-service-prod:5.15_130 not found: manifest unknown: manifest unknown
Apr 14 18:25:34 npl9app17 kubelet[207293]: E0414 18:25:34.142750  207293 pod_workers.go:191] Error syncing pod 022c4091-f8ce-4704-aa0c-1c36d70d63d0 ("abc-web-service-c49b95497-5sclr_99-content(022c4091-f8ce-4704-aa0c-1c36d70d63d0)"), skipping: failed to "StartContainer" for "abc-web-service" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: manifest for nexus.example.com/abc-web-service-prod:5.15_130 not found: manifest unknown: manifest unknown"
Apr 14 18:25:52 npl9app17 kubelet[207293]: E0414 18:25:52.406219  207293 remote_image.go:113] PullImage "nexus.example.com/abc-web-service-prod:5.15_130" from image service failed: rpc error: code = Unknown desc = Error response from daemon: manifest for nexus.example.com/abc-web-service-prod:5.15_130 not found: manifest unknown: manifest unknown
Apr 14 18:25:52 npl9app17 kubelet[207293]: E0414 18:25:52.406395  207293 kuberuntime_manager.go:803] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: manifest for nexus.example.com/abc-web-service-prod:5.15_130 not found: manifest unknown: manifest unknown
Apr 14 18:25:52 npl9app17 kubelet[207293]: E0414 18:25:52.406440  207293 pod_workers.go:191] Error syncing pod 022c4091-f8ce-4704-aa0c-1c36d70d63d0 ("abc-web-service-c49b95497-5sclr_99-content(022c4091-f8ce-4704-aa0c-1c36d70d63d0)"), skipping: failed to "StartContainer" for "abc-web-service" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: manifest for nexus.example.com/abc-web-service-prod:5.15_130 not found: manifest unknown: manifest unknown"

Upvotes: 0

Views: 436

Answers (1)

David Maze
David Maze

Reputation: 158908

My expectation would match yours. If the image is deleted on the upstream repository...

Running pods wouldn't be affected.

With imagePullPolicy: Always, any new pods will fail with ImagePullBackOff state.

With imagePullPolicy: IfNotPresent, any new pods that happen to get created on nodes that already have the image will succeed, and on nodes that don't have them image they will fail.

Note that in various cases pods can be created or deleted outside your immediate control. If a node is low on some resource, pods that are using more than their resource requests can get evicted; if you have the cluster autoscaler in use, it can terminate pods to free up a node to scale in the cluster; if you have a HorizontalPodAutoscaler configured, it can create and delete pods in response to load. One of these factors could cause the behavior you're seeing where a new pod is created (and failing to pull) without you taking explicit action.

Upvotes: 1

Related Questions