Azure AKS - pod keep CrashLoopBackOff status

Question

I am trying to deploy application from my personal docker registry into Azure AKS pods. I have python application that only logs some output:

import time
import logging

logger = logging.getLogger('main')
logger.setLevel(logging.INFO)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

def main():
    logger.info('This is test')
    time.sleep(5)


while True:
    try:
        main()
    except Exception:
        logger.critical('Something critical.', exc_info=1)

    logger.info('Sleep for 5 seconds')
    time.sleep(5)

And this is my Dockerfile:

FROM python:3.7-alpine

RUN apk update && apk upgrade

ARG APP_DIR=/app
RUN mkdir -p ${APP_DIR}
WORKDIR ${APP_DIR}

COPY requirements.txt .

RUN \
 apk add --no-cache --virtual .build-deps gcc python3-dev musl-dev linux-headers && \
 python3 -m pip install -r requirements.txt --no-cache-dir && \
 apk --purge del .build-deps

COPY app .

ENTRYPOINT [ "python", "-u", "run.py" ]

I am able to run the container on my local computer, here some logs:

docker logs -tf my-container
2020-02-07T10:26:57.939062754Z 2020-02-07 10:26:57,938 - main - INFO - This is test
2020-02-07T10:27:02.944500969Z 2020-02-07 10:27:02,943 - main - INFO - Sleep for 5 seconds
2020-02-07T10:27:07.948643749Z 2020-02-07 10:27:07,948 - main - INFO - This is test
2020-02-07T10:27:12.953683767Z 2020-02-07 10:27:12,953 - main - INFO - Sleep for 5 seconds
2020-02-07T10:27:17.955954057Z 2020-02-07 10:27:17,955 - main - INFO - This is test
2020-02-07T10:27:22.960453835Z 2020-02-07 10:27:22,959 - main - INFO - Sleep for 5 seconds
2020-02-07T10:27:27.964402790Z 2020-02-07 10:27:27,963 - main - INFO - This is test
2020-02-07T10:27:32.968647112Z 2020-02-07 10:27:32,967 - main - INFO - Sleep for 5 seconds

I am trying to deploy pod with this yaml file with kubectl apply -f onepod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: my-container
  labels:
    platform: xxx
    event: yyy
    protocol: zzz
spec:
    imagePullSecrets:
    - name: myregistry
    containers:
    - name: my-container
      image: mypersonalregistry/my-container:test

The pod is created but keep CrashLoopBackOff status without any output logs via kubectl logs command. I tried kubectl describe pod but in events are nothing usefull:

Name:         my-container
Namespace:    default
Priority:     0
Node:         aks-agentpool-56095163-vmss000000/10.240.0.4
Start Time:   Fri, 07 Feb 2020 11:41:48 +0100
Labels:       event=yyy
              platform=xxx
              protocol=zzz
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"event":"yyy","platform":"xxx","protocol":"zzz"},"name":"my-container...
Status:       Running
IP:           10.244.1.33
IPs:          
Containers:
  my-container:
    Container ID:   docker://c497674f86deadca2ef874f8a94361e26c770314e9cff1729bf20b5943d1a700
    Image:          mypersonalregistry/my-container:test
    Image ID:       docker-pullable://mypersonalregistry/my-container@sha256:c4208f42fea9a99dcb3b5ad8b53bac5e39bc54b8d89a577f85fec1a94535bc39
    Port:           
    Host Port:      
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 07 Feb 2020 12:28:10 +0100
      Finished:     Fri, 07 Feb 2020 12:28:10 +0100
    Ready:          False
    Restart Count:  14
    Environment:    
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-lv75n (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-lv75n:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-lv75n
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                                        Message
  ----     ------     ----                   ----                                        -------
  Normal   Scheduled  49m                    default-scheduler                           Successfully assigned default/my-container to aks-agentpool-56095163-vmss000000
  Normal   Pulled     48m (x5 over 49m)      kubelet, aks-agentpool-56095163-vmss000000  Container image "mypersonalregistry/my-container:test" already present on machine
  Normal   Created    48m (x5 over 49m)      kubelet, aks-agentpool-56095163-vmss000000  Created container my-container
  Normal   Started    48m (x5 over 49m)      kubelet, aks-agentpool-56095163-vmss000000  Started container my-container
  Warning  BackOff    4m55s (x210 over 49m)  kubelet, aks-agentpool-56095163-vmss000000  Back-off restarting failed container

How can I find out, why does it work on my computer but not in kubernetes cluster?

dorinand · Accepted Answer

So the porblem was with pulling latest version of my image. More here:

The default pull policy is IfNotPresent which causes the Kubelet to skip pulling an image if it already exists.

So it still run the first version of my-container with tag test and never download new one even when it is in my registry.

Solution is add this line to yaml file:

imagePullPolicy: Always

Azure AKS - pod keep CrashLoopBackOff status

Answers (2)

Related Questions