CODEWITHSUNDEEP
kubernetes
u123
u123

Reputation: 16287

High total CPU request but low total usage (kubernetes resources)

I have a bunch of pods in a cluster that is almost requesting all (7.35/8) available CPU resources on a node:

enter image description here

even though their actual total usage is almost nothing (0.34/8).

The pod that is currently requesting the most only requests 210m which I guess is not an outrageous amount - also I would like to enforce some sensible minimum request size for all pods in the cluster. Of course that will accumulate when there are lots of pods.

It seems I could easily scale down the request by a factor of 10 and leave the limits where they are to begin with.

But is there something else that I should look into instead before doing that - reducing replica count etc.?

Also it looks a bit strange that the pods are not more evenly distributed between the nodes.

enter image description here

Upvotes: 0

Views: 1967

Answers (3)

Wytrzymały Wiktor
Wytrzymały Wiktor

Reputation: 13878

There are good answers already but I would like to add some more info.

It is very important to have a good strategy when calculating how much resources you would need for each container. Optimally, your pods should be using exactly the amount of resources you requested but that's almost impossible to achieve. If the usage is lower than your request, you are wasting resources. If it's higher, you are risking performance issues. Consider a 25% margin up and down the request value as a good starting point. Regarding limits, achieving a good setting would depend on trying and adjusting. There is no optimal value that would fit everyone as it depends on many factors related to the application itself, the demand model, the tolerance to errors etc.

Kubernetes best practices: Resource requests and limits is a very good guide explaining the idea behind these mechanisms with a detailed explanation and examples.

Also, Managing Resources for Containers will provide you with the official docs regarding:

  • Requests and limits

  • Resource types

  • Resource requests and limits of Pod and Container

  • Resource units in Kubernetes

  • How Pods with resource requests are scheduled

  • How Pods with resource limits are run, etc

Just in case you'll need a reference.

Upvotes: 0

Narain
Narain

Reputation: 922

I suggest following:

  1. Resource Allocation: Based on history value set your request to meaningful value with buffer. Also to have guaranteed pod resource allocation it may be a good idea to set request and limit as same value. But that means you pod cannot burst for new resource. One more thing to note is scheduling only happens based on requested value, so if node has no more resource left, then pod will be killed and rescheduled if you request is trying to burst to limit.
  2. Resource quotas: Check Kubernetes Resource Quotas to have sensible namespace level quotas to control overly provisioned resources by developers
  3. Affinity/AntiAffinity: Check concept of Anti-affinity to have your replicas or different pods scheduled across your cluster. You can ensure for eg., that one host or Avalability zone etc can have only one replica of your pod (helps in HA), spread different pods to different nodes (layer scheduling etc) - Check this video

Upvotes: 0

XciD
XciD

Reputation: 2617

Your request values seems overestimated. You need time and metrics to find the right request/limit for your workload.

Keep in mind that if you change those values, your pods will restart.

Also, It's normal that you can find some unbalance nodes on your cluster. Kubernetes will never remove a pod if you don't ask.

For example, if your create a cluster with 3 nodes, fill those 3 nodes with pods and then add another 3 nodes. The new nodes will stay empty.

You can setup some HorizontalPodAutoScaler on your cluster to adapt your number of pod to your workload. Doing that, your workload will spread among nodes and with a correct balance. (if you use the default Scheduling Policy

Upvotes: 1

Related Questions