CODEWITHSUNDEEP
javalinuxlinux-kernelkubernetes
srkiNZ84
srkiNZ84

Reputation: 3568

How to get history of Pods run on Kubernetes Node?

In our Kuberenetes cluster, we are running into sporadic situations where a cluster node runs out of memory and Linux invokes OOM killer. Looking at the logs, it appears that the Pods scheduled onto the Node are requesting more memory than can be allocated by the Node.

The issue is that, when OOM killer is invoked, it prints out a list of processes and their memory usage. However, as all of our Docker containers are Java services, the "process name" just appears as "java", not allowing us to track down which particular Pod is causing the issues.

How can I get the history of which Pods were scheduled to run on a particular Node and when?

Upvotes: 14

Views: 40695

Answers (5)

Aydin K.
Aydin K.

Reputation: 3367

Event history for your particular namespace, ordered by creationTimestamp:

kubectl get events -n YOURNAMESPACE -o wide --sort-by=.metadata.creationTimestamp

Or if you want to check the event history for all namespaces, ordered by creationTimestamp:

kubectl get events --all-namespaces -o wide --sort-by=.metadata.creationTimestamp

Upvotes: 3

Camil
Camil

Reputation: 8396

You can now use kube-state-metrics kube_pod_container_status_terminated_reason to detect OOM events

kube_pod_container_status_terminated_reason{reason="OOMKilled"}

kube_pod_container_status_terminated_reason{container="addon-resizer",endpoint="http-metrics",instance="100.125.128.3:8080",job="kube-state-metrics",namespace="monitoring",pod="kube-state-metrics-569ffcff95-t929d",reason="OOMKilled",service="kube-state-metrics"}

Upvotes: 10

Paweł Prażak
Paweł Prażak

Reputation: 3191

We use Prometheus to monitor OOM events.

This expression should report the number of times that memory usage has reached the limits:

rate(container_memory_failcnt{pod_name!=""}[5m]) > 0

FYI: this is the next best thing to proper docs, the code

Upvotes: 5

caarlos0
caarlos0

Reputation: 20633

I guess your pods don't have requests and limits set, or the values are not ideal.

If you setup this properly, when a pod starts to use too much ram, that pod will be killed and you will be able to find out what is causing the issues.

About seeing all the pods on a node, you can go with kubectl get events or docker ps -a on the node, as cited on the other answers/comments.

Upvotes: 0

manojlds
manojlds

Reputation: 301117

One way is to see docker ps -a output and correlate the container names with your pod's containers.

Upvotes: -4

Related Questions