CODEWITHSUNDEEP
kubernetes
Shashank Omre
Shashank Omre

Reputation: 301

Handle Kubernetes pod exceeding CPU resource limit?

NAME                      CPU(cores)   MEMORY(bytes)   
apache-757ddfbc75-khhfw   10m          61Mi

Upvotes: 14

Views: 17832

Answers (5)

Ostrava
Ostrava

Reputation: 101

One of many things can and will happen from my experience:

  1. Pod will log OOM and crash / restart. This scenario usually happens when requests are made to the pod. If users are having issues, check how many times the pod restarted and see if there is a correlation between requests to the service and restarts.
  2. If Azure Policy is being enforced (when using it), the pods will simply not be created / will shut down.
  3. Deployments will not succeed.

Upvotes: -1

Cristian
Cristian

Reputation: 733

It seems throttling occurs (this is NOT written in the official docs):

Kubernetes uses kernel throttling to implement CPU limit. If an application goes above the limit, it gets throttled (aka fewer CPU cycles). Memory requests and limits, on the other hand, are implemented differently, and it’s easier to detect. You only need to check if your pod’s last restart status is OOMKilled.

https://medium.com/omio-engineering/cpu-limits-and-aggressive-throttling-in-kubernetes-c5b20bd8a718#:~:text=Kubernetes%20uses%20kernel%20throttling%20to,last%20restart%20status%20is%20OOMKilled

Upvotes: 5

FakeAlcohol
FakeAlcohol

Reputation: 990

k8s doc refer:

A Container might or might not be allowed to exceed its CPU limit for extended periods of time. However, it will not be killed for excessive CPU usage.

Most of those case, nothing will happen. CPU usage is very very flexible.

Upvotes: 9

Malgorzata
Malgorzata

Reputation: 7023

Starting from scratch

When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled Containers is less than the capacity of the node. Note that although actual memory or CPU resource usage on nodes is very low, the scheduler still refuses to place a Pod on a node if the capacity check fails. This protects against a resource shortage on a node when resource usage later increases, for example, during a daily peak in request rate.

To specify a CPU request for a container, include the resources:requests field in the Container resource manifest. To specify a CPU limit, include resources:limits.

CPU requests and limits are associated with Containers, but it is useful to think of a Pod as having a CPU request and limit. The CPU request for a Pod is the sum of the CPU requests for all the Containers in the Pod. Likewise, the CPU limit for a Pod is the sum of the CPU limits for all the Containers in the Pod.

Pod scheduling is based on requests. A Pod is scheduled to run on a Node only if the Node has enough CPU resources available to satisfy the Pod CPU request.

In below pod's configuration file the container requests 100 CPU, which is likely to exceed the capacity of any Node in your cluster.

apiVersion: v1
kind: Pod
metadata:
  name: cpu-demo
  namespace: cpu-test
spec:
  containers:
  - name: cpu-demo-ctr-2
    image: vish/stress
    resources:
      limits:
        cpu: "100"
      requests:
        cpu: "100"
    args:
    - -cpus
    - "2"

After creating pod you will get similar output:

enter image description here

It shows that the container cannot be scheduled because of insufficient CPU resources on the Nodes.

Overall documentations: kubernetes-resources, managing-resources.

Upvotes: -1

Keilo
Keilo

Reputation: 986

Docs: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run

"A Container might or might not be allowed to exceed its CPU limit for extended periods of time. However, it will not be killed for excessive CPU usage."

Container will not be allowed to use more CPU than it's limit on average, other container will be protected from excessive CPU usage.

Upvotes: -1

Related Questions