Joey Yi Zhao
Joey Yi Zhao

Reputation: 42576

why does my k8s cluster reports node `overcommitted.`?

I deploy my application to AWS EKS cluster with 3 nodes. When I run describe, it shows me this message: (Total limits may be over 100 percent, i.e., overcommitted.). But based on the full message, it doesn't seem there are many resources. Why do i see this message in the output?

$ kubectl describe node ip-192-168-54-184.ap-southeast-2.compute.internal
Name:               ip-192-168-54-184.ap-southeast-2.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=t3.medium
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/capacityType=ON_DEMAND
                    eks.amazonaws.com/nodegroup=scalegroup
                    eks.amazonaws.com/nodegroup-image=ami-0ecaff41b4f38a650
                    failure-domain.beta.kubernetes.io/region=ap-southeast-2
                    failure-domain.beta.kubernetes.io/zone=ap-southeast-2b
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-192-168-54-184.ap-southeast-2.compute.internal
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=t3.medium
                    topology.kubernetes.io/region=ap-southeast-2
                    topology.kubernetes.io/zone=ap-southeast-2b
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 04 Mar 2021 22:27:50 +1100
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  ip-192-168-54-184.ap-southeast-2.compute.internal
  AcquireTime:     <unset>
  RenewTime:       Fri, 05 Mar 2021 09:13:16 +1100
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 05 Mar 2021 09:11:33 +1100   Thu, 04 Mar 2021 22:27:50 +1100   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 05 Mar 2021 09:11:33 +1100   Thu, 04 Mar 2021 22:27:50 +1100   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 05 Mar 2021 09:11:33 +1100   Thu, 04 Mar 2021 22:27:50 +1100   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 05 Mar 2021 09:11:33 +1100   Thu, 04 Mar 2021 22:28:10 +1100   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   192.168.54.184
  ExternalIP:   13.211.200.109
  Hostname:     ip-192-168-54-184.ap-southeast-2.compute.internal
  InternalDNS:  ip-192-168-54-184.ap-southeast-2.compute.internal
  ExternalDNS:  ec2-13-211-200-109.ap-southeast-2.compute.amazonaws.com
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         2
  ephemeral-storage:           20959212Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      3970504Ki
  pods:                        17
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         1930m
  ephemeral-storage:           18242267924
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      3415496Ki
  pods:                        17
System Info:
  Machine ID:                 ec246b12e91dc516024822fbcdac4408
  System UUID:                ec246b12-e91d-c516-0248-22fbcdac4408
  Boot ID:                    5c6a3d95-c82c-4051-bc90-6e732b0b5be2
  Kernel Version:             5.4.91-41.139.amzn2.x86_64
  OS Image:                   Amazon Linux 2
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.6
  Kubelet Version:            v1.19.6-eks-49a6c0
  Kube-Proxy Version:         v1.19.6-eks-49a6c0
ProviderID:                   aws:///ap-southeast-2b/i-03c0417efb85b8e6c
Non-terminated Pods:          (4 in total)
  Namespace                   Name                                     CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                     ------------  ----------  ---------------  -------------  ---
  cert-manager                cert-manager-cainjector-9747d56-qwhjw    0 (0%)        0 (0%)      0 (0%)           0 (0%)         10h
  kube-system                 aws-node-m296t                           10m (0%)      0 (0%)      0 (0%)           0 (0%)         10h
  kube-system                 coredns-67997b9dbd-cgjdj                 100m (5%)     0 (0%)      70Mi (2%)        170Mi (5%)     10h
  kube-system                 kube-proxy-dc5fh                         100m (5%)     0 (0%)      0 (0%)           0 (0%)         10h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests    Limits
  --------                    --------    ------
  cpu                         210m (10%)  0 (0%)
  memory                      70Mi (2%)   170Mi (5%)
  ephemeral-storage           0 (0%)      0 (0%)
  hugepages-1Gi               0 (0%)      0 (0%)
  hugepages-2Mi               0 (0%)      0 (0%)
  attachable-volumes-aws-ebs  0           0
Events:                       <none>

Upvotes: 0

Views: 1439

Answers (1)

matt_j
matt_j

Reputation: 4614

Let's quickly analyze the source code of the kubectl describe command, in particular the describeNodeResource function.

Inside the describeNodeResource(...) function we see ( this line ):

w.Write(LEVEL_0, "Allocated resources:\n  (Total limits may be over 100 percent, i.e., overcommitted.)\n")

There is no condition to check when this message should be printed, it is just an informational message that is printed every time.

Upvotes: 1

Related Questions