CODEWITHSUNDEEP
kubernetesgoogle-kubernetes-enginekubernetes-podevict
JorgeeFG
JorgeeFG

Reputation: 5981

If requested memory is "the minimum", why is kubernetes killing my pod when it exceeds 10x the requested?

I am debuggin a problem with pod eviction in Kubernetes.

It looks like it is related to a configuration in PHP FPM children processes quantity.

I assigned a minimum memory of 128 MB and Kubernetes is evicting my pod apparently when exceeds 10x that amount (The node was low on resource: memory. Container phpfpm was using 1607600Ki, which exceeds its request of 128Mi.)

How can I prevent this? I thought that requested resources is the minimum and that the pod can use whatever is available if there's no upper limit.

Upvotes: 1

Views: 2134

Answers (1)

Andrew
Andrew

Reputation: 4642

Requested memory is not "the minimum", it is what it is called - the amount of memory requested by pod. When kubernetes schedules pod, it uses request as a guidance to choose a node which can accommodate this workload, but it doesn't guarantee that pod won't be killed if node is short on memory.

As per docs https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-limits-are-run

if a container exceeds its memory request and the node that it runs on becomes short of memory overall, it is likely that the Pod the container belongs to will be evicted.

If you want to guarantee a certain memory window for your pods - you should use limits, but in that case if your pod doesn't use most of this memory, it will be "wasted"

So to answer your question "How can I prevent this?", you can:

  • reconfigure your php-fpm in a way, that prevents it to use 10x memory (i.e. reduce workers count), and configure autoscaling. That way your overloaded pods won't be evicted, and kubernetes will schedule new pods in event of higher load
  • set memory limit to guarantee a certain amount of memory to your pods
  • Increase memory on your nodes
  • Use affinity to schedule your demanding pods on some dedicated nodes and other workloads on separate nodes

Upvotes: 4

Related Questions