CODEWITHSUNDEEP
kuberneteskubectldisk-access
bskaggs
bskaggs

Reputation: 1504

How can I determine the current ephemeral-storage usage of a running Kubernetes pod?

How can I tell with kubectl how much ephemeral storage a pod is currently using?

In a Kubernetes pod spec, I can specify resource requests and limits for CPU, memory, and ephemeral storage:

resources:
  requests:
    memory: "60Mi"
    cpu: "70m"
    ephemeral-storage: "2Gi"
  limits:
    memory: "65Mi"
    cpu: "75m"
    ephemeral-storage: "4Gi"

However, to set good requests and limits on ephemeral storage, I need to know what this value actually is for a running pod, which I can't figure out. I can get CPU and memory usage using kubectl top pod, but, from what I can tell, ephemeral storage usage is only actually calculated when making an actual eviction decision.

Upvotes: 44

Views: 58473

Answers (2)

jmcgrath207
jmcgrath207

Reputation: 2087

You can do this through the raw command.

kubectl get --raw "/api/v1/nodes/(your-node-name)/proxy/stats/summary"

There is also this

kubectl get --raw "/api/v1/nodes/(your-node-name)/proxy/metrics/cadvisor"

EDIT:

I've created a Prometheus exporter for this now.

https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics

Upvotes: 28

Rico
Rico

Reputation: 61689

The pure raw approach for this is to use the disk usage (du) Unix command line.

Shell into your pod:

$ kubectl exec -it <pod-id> sh

Change dirs to the mount point of your ephemeral-storage (if you are using volume mounts):

$ mount # check mount points if you'd like
$ cd /mnt/of/ephemeral
$ du .

If you are not using volume mounts:

$ du .

There are other tools that you can use to get metrics:

  • cAdvisor also embedded into the kubelet code, exposed under the /stats/summary or /metrics endpoint. More info here. An example output:

    $ curl -k -H 'Authorization: Bearer <Redacted>' \
https://node-hostname:10250/stats/summary

{
 "node": {
   "nodeName": "node-hostname",
   "systemContainers": [
    {
     "name": "kubelet",
    ...
    "volume": [
     {
      "time": "2018-11-08T23:52:03Z",
      "availableBytes": 1969168384,
      "capacityBytes": 1969180672,
      "usedBytes": 12288,
      "inodesFree": 480748,
      "inodes": 480757,
      "inodesUsed": 9,
      "name": "kube-proxy-token-pprwb"
     }
    ],
    "ephemeral-storage": {
     "time": "2018-11-09T00:05:10Z",
     "availableBytes": 31057477632,
     "capacityBytes": 41567858688,
     "inodesFree": 4873887,
     "inodes": 5120000
    }
...
}

    Similarly:

    $ curl -k -H 'Authorization: Bearer <Redacted>' \
https://node-hostname:10250/stats/summary

# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
...

    More info on kubelet authentication/authorization.

  • Prometheus

More info on K8s metrics here.

Upvotes: 9

Related Questions