CODEWITHSUNDEEP
kubernetesgoogle-cloud-platform
Ragnar
Ragnar

Reputation: 2690

Pods evicted due to memory or OOMKilled

New to kubernetes I try to start a K8S (on GCP) pod with airflow. The first runs terminated as "evicted"

The node was low on resource: memory. Container base was using 5168120Ki, which exceeds its request of 0.

I don't understand the end of this message. It exceed by 0 ?

Then I started to read the documentation and learn about request and limit CPU and memory. I have run my Python program on my machine and see that I need around 800Mo of RAM. I then decided to set the request = 1G and limit = 2G. With that set I have another error OOMKilled.

I even try 3G but I got the same. My cluster is with 2 nodes, 4 vCPU, 15Gb Ram.

Why do I have evicted in the first case and OOM in the second ?

Upvotes: 5

Views: 12971

Answers (1)

David Maze
David Maze

Reputation: 158656

Given that error message, you need to set the memory limit to at least 5 GiB. (And if you're surprised by that, you might try to replicate your workload outside Kubernetes with a heap profiler to see what's going on.)

In the first situation, the node itself is running out of memory (all of that 15 Gb is in use by some set of processes). Here the Kubernetes scheduler looks at the pods running on the node and finds ones that are significantly over their resource requests. The error message says you're using 5 GiB more memory than you requested, and you didn't request anything, so you get evicted. The pod should get scheduled on a different node.

In the second situation, you've limited the amount of memory that can be used, to 2 or 3 GiB. From the first error message, though, the process wants to use 5 GiB. When you reach that configured limit, you get OOMKilled (out-of-memory) and will get restarted.

Setting these limits correctly is a little bit of an art. The first thing to know is how much memory your process actually uses. If you can run it offline, basic tools like top or ps can tell you this; if you have Kubernetes metrics set up, a monitoring tool like Prometheus can also identify per-pod memory use. You need to set the memory limit to at least the largest amount of memory you expect the pod to use (or else it gets OOM-killed).

Memory requests are less scientific. If the node has 15 GiB of RAM, the scheduler will allow pods with a total of 15 GiB of requested memory to run there. If you set resource requests close to or equal to resource limits, then fewer other pods will be able to run on the node; the node has less chance of running out of memory; the individual pod has less chance of getting evicted; and if pods in aggregate aren't using all the memory they requested then some resource will be unused. Setting a lower resource request allows more pods to run on the node and higher physical resource utilization (and potentially fewer nodes), but if that goes above 100% things will get evicted.

I tend to set resource requests around a process's typical use, from the same monitoring you used to find the limit. I've also seen very reasonable suggestions to set requests and limits the same to avoid pod eviction. It depends on what your tolerance for pods getting restarted is, compared to your tolerance to run (and pay for) additional nodes with lower utilization.

Upvotes: 6

Related Questions