CODEWITHSUNDEEP
google-compute-enginegoogle-cloud-stackdriver
AntsaR
AntsaR

Reputation: 390

How to track disk usage on Container-Optimized OS

I have an application running on Container-Optimized OS based Compute Engine. My application runs every 20min, fetches and writes data to a local file, then deletes the file after some processing. Note that each file is less than 100KB. My boot disk size is the default 10GB. I run into "no space left on device" error every month or so while attempting to write the file locally.

How can I track disk usage?

I manually checked the size of the folders and it seems that the bulk of the space is taken by /mnt/stateful_partition/var/lib/docker/overlay2. 

my-vm / # sudo du -sh /mnt/stateful_partition/var/lib/docker/*
20K     /mnt/stateful_partition/var/lib/docker/builder
72K     /mnt/stateful_partition/var/lib/docker/buildkit
208K    /mnt/stateful_partition/var/lib/docker/containers
4.4M    /mnt/stateful_partition/var/lib/docker/image
52K     /mnt/stateful_partition/var/lib/docker/network
1.6G    /mnt/stateful_partition/var/lib/docker/overlay2
20K     /mnt/stateful_partition/var/lib/docker/plugins
4.0K    /mnt/stateful_partition/var/lib/docker/runtimes
4.0K    /mnt/stateful_partition/var/lib/docker/swarm
4.0K    /mnt/stateful_partition/var/lib/docker/tmp
4.0K    /mnt/stateful_partition/var/lib/docker/trust
28K     /mnt/stateful_partition/var/lib/docker/volumes

Upvotes: 1

Views: 2325

Answers (2)

Armando Cuevas
Armando Cuevas

Reputation: 889

TL;DR: Use Stackdriver Monitoring and create an alert for DISK usage.

Since you are using COS images, you can enable Stackdriver Monitoring agent by simply adding the “google-monitoring-enabled” label set to “true” on GCE Instance metadata. To do so, run the command:

gcloud compute instances add-metadata instance-name --metadata=google-monitoring-enabled=true

Replace instance-name with the name of your instance. Remember to restart your instance to get the change done. You don't need to install the Stackdriver Monitoring agent since is already installed by default in COS images.

Then, you can use disk usage metric to get the usage of your disk. How to get DISK USAGE in Stackdriver Monitoring

You can create an alert to get a notification each time the usage of the partition reaches a certain threshold.

Since you are in a cloud, it is always the best idea to use the Cloud resources to solve Cloud issues.

Upvotes: 3

William
William

Reputation: 151

Docker uses /var/lib/docker to store your images, containers, and local named volumes. Deleting this can result in data loss and possibly stop the engine from running. The overlay2 subdirectory specifically contains the various filesystem layers for images and containers.

To cleanup unused containers and images via command: docker system prune.

Monitor it via command "watch" sudo watch "du -sh /mnt/stateful_partition/var/lib/docker/*"

Upvotes: 0

Related Questions