CODEWITHSUNDEEP
kubernetes
Nathan Long
Nathan Long

Reputation: 126122

How can I keep a Pod from crashing so that I can debug it?

In Kubernetes, when a Pod repeatedly crashes and is in CrashLoopBackOff status, it is not possible to shell into the container and poke around to find the problem, due to the fact that containers (unlike VMs) live only as long as the primary process. If I shell into a container and the Pod is restarted, I'm kicked out of the shell.

How can I keep a Pod from crashing so that I can investigate if my primary process is failing to boot properly?

Upvotes: 11

Views: 11064

Answers (3)

Aram
Aram

Reputation: 716

in my case I did build using mac m1/silicon. In this case pod crashes and there is no explicit message about this.

The problem was that I a also debugged using docker on the same m1 so could not really see what is wrong.

I would need to build image using docker build --platform linux/amd64.

Upvotes: 1

Wytrzymały Wiktor
Wytrzymały Wiktor

Reputation: 13928

There are also other methods used for debugging pods that are worth noting in your use case scenario:

  • If your container has previously crashed, you can access the previous container's crash log with: kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}

  • Debugging with an ephemeral debug container: Ephemeral containers are useful for interactive troubleshooting when kubectl exec is insufficient because a container has crashed or a container image doesn't include debugging utilities, such as with distroless images. kubectl has an alpha command that can create ephemeral containers for debugging beginning with version v1.18. An example for this method can be found here.

Upvotes: 2

Nathan Long
Nathan Long

Reputation: 126122

Redefine the command

In development only, a temporary hack to keep a Kubernetes pod from crashing is to redefine it and specify the container's command (corresponding to a Docker ENTRYPOINT) and args to be a command that will not crash. For instance:

  containers:
  - name: something
    image: some-image
    # `shell -c` evaluates a string as shell input
    command: [ "sh", "-c"]
    # loop forever, outputting "yo" every 5 seconds
    args: ["while true; do echo 'yo' && sleep 5; done;"]

This allows the container to run and gives you a chance to shell into it, like kubectl exec -it pod/some-pod -- sh, and investigate what may be wrong.

This needs to be undone after debugging so that the container will run the command it's actually meant to run.

Adapted from this blog post.

Upvotes: 16

Related Questions