CODEWITHSUNDEEP
kubernetes
pyron_orion
pyron_orion

Reputation: 635

Unable to set LimitRange: Creating Pods is forbidden[ maximum cpu usage is x but limit is y...]

I have 3 delpoyments in my namespace. None of the deployments specify any resource limits for either CPU or memory. I am trying to set the default min and max using LimitRange for all pods and their containers(existing and future) in this namespace. I've deployed a LimitRange resource to the namespace as defined below. However, when I redeploy, the deployments fail with errors (as listed below)

LimitRange:

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "core-resource-limits"
  namespace: x
spec:
  limits:
    - type: "Pod"
      max:
        cpu: 1
        memory: 1Gi
      min:
        cpu: 2m
        memory: 50Mi
    - type: "Container"
      max:
        cpu: 1
        memory: 800Mi
      min:
        cpu: 1m
        memory: 50Mi
      default:
        cpu: 20m
        memory: 500Mi
      defaultRequest:
        cpu: 10m
        memory: 400Mi
      maxLimitRequestRatio:
        cpu: 4

Error This is the error that I see

I don't understand where is 2040m coming from as its not defined in the LimitRange and its not defined in any of the deployments. Similarly all the other limit values. I have tried changing all these values to a bit higher/lower but I can't figure it out. Can someone explain what is wrong with this set of values?

Thanks

Edit: All the pods are on the same node. The node limit are as follows:

  1. CPU requests: 2.86
  2. CPU limits: 48.55
  3. Memory requests: 7.459
  4. Memory limits: 28.342

Upvotes: 4

Views: 6037

Answers (2)

Wytrzymały Wiktor
Wytrzymały Wiktor

Reputation: 13858

I would like to expand and explain in more detail what might be wrong here. Let's take a closer look at the overview of Limit Range:

  • The administrator creates one LimitRange in one namespace.

  • Users create resources like Pods, Containers, and PersistentVolumeClaims in the namespace.

  • The LimitRanger admission controller enforces defaults and limits for all Pods and Containers that do not set compute resource requirements and tracks usage to ensure it does not exceed resource minimum, maximum and ratio defined in any LimitRange present in the namespace.

  • If creating or updating a resource (Pod, Container, PersistentVolumeClaim) that violates a LimitRange constraint, the request to the API server will fail with an HTTP status code 403 FORBIDDEN and a message explaining the constraint that have been violated.

  • If a LimitRange is activated in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values. Otherwise, the system may reject Pod creation.

  • LimitRange validations occurs only at Pod Admission stage, not on Running Pods.

As already mentioned by @FritzDuchardt the error message clearly states that the limits are misconfigured or wrongly enforced. This leads us to two ways:

  • Check if there are any other limits set at the Pod level (kubectl edit pod <pod_name>).

  • Within a namespace, a Pod or Container can consume as much CPU and memory as defined by the namespace's resource quota. Check if resource quotas are set with kubectl describe quota. In the case where the total limits of the namespace is less than the sum of the limits of the Pods/Containers, there may be contention for resources. In this case, the Containers or Pods will not be created.

Here is an example of attempting to create a Pod that exceeds the maximum memory constraint which represents the same result as you are experiencing.

I hope this explains the topic and potential issues in more detail. Please let me know if that helps.

Upvotes: 3

Fritz Duchardt
Fritz Duchardt

Reputation: 11860

Looking at the error message, I would say your Pod resources limits are set higher than permitted by the LimitRange. Do a "kubectl edit" on the exact Pod in question to ensure there are not limits set.

Upvotes: 0

Related Questions