CODEWITHSUNDEEP
kubernetesprometheuskubectl
adebasi
adebasi

Reputation: 3945

How to get container's disk usage in Kubernetes (without docker command)?

I have a Kubernetes Cluster and want to know how much disk space my containers use. I am not talking about mounted Volumes.

I can get this information by using docker commands like docker system df -v or docker ps -s, but I don't want to connect to every single worker node.

Is there a way to get a container's disk usage via kubectl or are there kubelet metrics where I can get this information from?

Upvotes: 11

Views: 20004

Answers (4)

Eugene Bykov
Eugene Bykov

Reputation: 1

As a one more addition which uses kubectl instead of curl and prints it as a table. It also prints logs size

echo -e "NODE\tNAMESPACE\tPOD\tCONTAINER\tROOTFS\tLOGS"
kubectl get --raw "/api/v1/nodes/${node}/proxy/stats/summary" | jq -r '. | .node.nodeName as $node | .pods[] | .podRef.namespace as $namespace | .podRef.name as $pod | .containers[] | [$node, $namespace, $pod, .name, .rootfs.usedBytes, .logs.usedBytes] | @tsv'

Upvotes: 0

Charlie Large
Charlie Large

Reputation: 81

To add on to Stephen's answer, here is a curl query if you want the pod name alongside the size in bytes.

curl http://localhost:8080/api/v1/nodes/<node-name>/proxy/stats/summary 2>/dev/null | jq '.pods[].containers[] | "\(.name) \(.rootfs.usedBytes)"'

Upvotes: 2

Stephen Paulger
Stephen Paulger

Reputation: 5343

Adding to Rico's answer you can use kubectl to set up a proxy to the API server so that you can query the API with the details of the server address and authentication taken care of.

Set up a local proxy to the API server with

kubectl proxy --port=8080

Then you can use a command like

curl http://localhost:8080/api/v1/nodes/<node-name>/proxy/stats/summary 2>/dev/null | jq '.pods[].containers[].rootfs.usedBytes'

which will give you a list of the rootfs usage by each container.

Upvotes: 5

Rico
Rico

Reputation: 61641

Yes, but currently not with kubectl, you can get metrics from the kubelet, either through the kube-apiserver (proxied) or directly calling the kubelet HTTP(s) server endpoint (default port 10250). Disk metrics are generally available on the /stats/summary endpoint and you can also find some cAdvisor metrics on the /metrics/cavisor endpoint.

For example, to get the 'usedBytes' for the first container in the first pod returned through the kube-apiserver:

$ curl -k -s -H 'Authorization: Bearer <REDACTED>' \
  https://kube-apiserver:6443/api/v1/nodes/<node-name>/proxy/stats/summary \
  | jq '.pods[0].containers[0].rootfs.usedBytes'

The Bearer token can be a service account token tied to a ClusterRole like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
  name: myrole
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  - /api/*
  verbs:
  - get

Upvotes: 13

Related Questions