CODEWITHSUNDEEP
kubernetes
Matheus Neder
Matheus Neder

Reputation: 139

How to track Kubernetes OOMKilled termination messages?

When a Pod is evicted I'm supposed to find messages about the eviction at /dev/termination-log (or other path defined in terminationMessagePath property) inside the container filesystem.

I'm trying to track OOMKilled events, but after OOMKilled occured and the Pod was restarted, the /dev/termination-log is empty.
I tried to set terminationMessagePolicy property to FallbackToLogsOnError hoping to find out messages in container log and I did not find anything about the OOMKilled there.

By executing kubectl describe pod podname I can find out that the termination reason was OOMKilled, but I would like to be able to follow this events in order to integrate it with fluentd and logstash.

Our Kubernetes cluster version is v1.9.0 and it's running on-premise. The installation was done using kubeadm on top of CentOS 7.

Upvotes: 10

Views: 20278

Answers (3)

Oana
Oana

Reputation: 657

It might not return the error code, but it certainly shows more details.

You can use kubectl logs --previous to retrieve logs from a previous instantiation of a container. If your pod has multiple containers, specify which container's logs you want to access by appending a container name to the command.

from https://kubernetes.io/docs/concepts/cluster-administration/logging/

Upvotes: 2

Кирилл Семаев
Кирилл Семаев

Reputation: 41

The way I'm using - is looking to dmesg on nodes and track events from there

Upvotes: 3

Cindy
Cindy

Reputation: 272

You can ssh to the machine the pod was running and exec journalctl -u kubelet

Upvotes: 5

Related Questions