CODEWITHSUNDEEP
kubernetesazure-aks
Neo
Neo

Reputation: 468

Kubectl top delivers different results between node and pods

I noticed that the amount of consumed resources claimed by the top pods -A command does not match the one shown by the top node one (intended as the sum of pods on a specific node).

In the specific, for a single node I see a difference of about 1Gi in memory consumption, which to me it is quite a lot. I also tried to query directly the results from the metrics API, which confirms the difference.

Does anyone know why and how could it be possible? Does the top node command account also for not running resources? If so, why?

My cluster is running on an Azure AKS managed instance, and I see the same problem both on Linux and Windows nodes.

Upvotes: 0

Views: 1379

Answers (2)

Jeff R
Jeff R

Reputation: 645

I thought I was experiencing a discrepancy also, but after I realized that kubectl top pod --all-namespaces supports a --sum parameter, it made it easier to see that I was just bad at math and getting an appropriate number!


kubectl top pods --all-namespaces --sum  
NAMESPACE         NAME                                          CPU(cores)   MEMORY(bytes)   
ambassador        edge-stack-794f7bc6f9-dc9sz                   19m          188Mi           
ambassador        edge-stack-794f7bc6f9-dnz8c                   15m          231Mi           
ambassador        edge-stack-794f7bc6f9-ql47q                   19m          224Mi           
ambassador        edge-stack-agent-564f89667d-mll6r             3m           34Mi            
ambassador        edge-stack-redis-f4d8d6b5d-ljmdw              5m           9Mi             
emissary-system   emissary-apiext-58565b6f5c-h2c2l              2m           27Mi            
emissary-system   emissary-apiext-58565b6f5c-krsdg              1m           34Mi            
emissary-system   emissary-apiext-58565b6f5c-ql8p8              3m           27Mi            
kube-system       cilium-gtf6m                                  21m          115Mi           
kube-system       cilium-operator-599bbfc6cc-cp5lr              3m           32Mi            
kube-system       cilium-rjfvj                                  16m          87Mi            
kube-system       coredns-575d7877bb-4pbtg                      3m           27Mi            
kube-system       coredns-575d7877bb-sqqrg                      3m           23Mi            
kube-system       cpc-bridge-proxy-lg7h4                        1m           2Mi             
kube-system       cpc-bridge-proxy-zzk6d                        1m           2Mi             
kube-system       csi-do-node-94szc                             1m           22Mi            
kube-system       csi-do-node-kr8qm                             1m           25Mi            
kube-system       do-node-agent-j7c74                           1m           30Mi            
kube-system       do-node-agent-mjwrz                           1m           26Mi            
kube-system       hubble-relay-67b5f94fb5-qfdvp                 1m           16Mi            
kube-system       hubble-ui-79db4488b7-qddj4                    0m           23Mi            
kube-system       konnectivity-agent-4l4xn                      4m           19Mi            
kube-system       konnectivity-agent-qbvxt                      3m           20Mi            
kube-system       kube-proxy-btp5p                              1m           29Mi            
kube-system       kube-proxy-jvbll                              1m           31Mi            
metrics-server    metrics-server-6cb95bfdb6-n4hx2               4m           20Mi            
ts-ops            ts-ops-backend-dep-7977cc6f9b-vmsrp           62m          168Mi           
                                                                ________     ________        
                                                                181m         1505Mi

Upvotes: 1

Sahadat Hossain
Sahadat Hossain

Reputation: 4349

kubectl top node: It displays resource (CPU/Memory/Storage) usage of nodes.

Basically, reserve a portion of the CPU and memory resources for use by the underlying node components such as kubelet, kube-proxy, and the container engine. And it reserves that portion in the first time when you configured the node, then that node divides those memory to the pods or k8s resources according to need.

You can see this doc for more info.

Upvotes: 1

Related Questions