CODEWITHSUNDEEP
dockerdevops
Nikola Roganovic
Nikola Roganovic

Reputation: 83

Docker overlay2 folder eating Disk

i have problem with docker overlay2 folder.

When I enter du -sj /var/lib/docker/overlay2 in the time of writing it says:

85G overlay2/

And it keeps growing up. I tried to prune system with command like:

docker system prune -a

But output is :

Total reclaimed space: 0B

I also tried to add max-file and max-size to my docker compose. After adding these lines I recreated containers. But problem is still there, can it be solved in any way, just to stop eating disk space ?

EDIT:

docker info:

Server:
Containers: 10
Running: 10
Paused: 0
Stopped: 0
Images: 6
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem:
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local \logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-91-generic
Operating System: Ubuntu 18.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 62.86GiB
Name: machine
ID: TRX2:AJZG:LNJF:UCPW:MQQB:2PLR:R7KM:VHEV:KBOG:IBT4:JX4R:WR6V
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/\ Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false\

docker image: (for the security reasons i wont show the names)

TAG IMAGE ID CREATED SIZE
latest 667ac029b200 2 years ago 7.6GB
latest cb9df5aac4ac 2 years ago 12.9GB
latest d445c0adc9a5 3 weeks ago 220MB
latest 227d9f0554d3 2 years ago 1.34GB
latest b80092729008 2 years ago 758MB
latest f32a97de94e1 2 years ago 25.8MB

Upvotes: 5

Views: 11594

Answers (4)

not2savvy
not2savvy

Reputation: 4243

I've made the same experience, and it seems like docker prune --all doesn't (always? at all?) work as expected.

What helped me:

Check current usage before we begin

   docker system df

  1. Remove all containers older than 35 days (adjust to your liking)

    docker container prune --filter "until=840h" --force

  2. Remove unused volumes

    docker volume prune --force

  3. Remove dangling volumes (docker system prune should actually take care of this, but often doesn't)

    docker volume rm $(docker volume ls -q --filter dangling=true)

  4. Finally, delete all cache files that are older than 48 hours

    docker builder prune --all --filter until=48h --force

Check again to see the results

docker system df

Upvotes: 0

BMitch
BMitch

Reputation: 263896

You have 10 containers running, and the 2 year old images are a strong indication that you're likely mutating the filesystem inside those containers (writing logs, temp files, maybe even installing apps). A prune will not delete those containers, their images, the filesystem changes within those containers, or the logs of the containers. The part that you'll see in overlay2 are those filesystem changes, so use the following to see what files have been created/changed within each container:

docker container diff ${container_name_or_id}

e.g.:

$ docker container diff d7c
C /run
A /run/user
A /run/user/1000
C /tmp
A /tmp/.X11-unix
C /home
A /home/user

Your list will likely be longer, and you can go into the container to delete any files created that you no longer need (using docker exec assuming the container has a shell and other CLI tools).

For more on cleaning up a large overlay2 directory, see is it safe to clean docker/overlay2.

Upvotes: 1

Shaktirajsinh Jadeja
Shaktirajsinh Jadeja

Reputation: 409

  1. Check with docker volume ls , it will show volumes and size
  2. whatever not required clear them
  3. If you are using linux based images you can use - apt-get autoremove , to remove orphan packages

Upvotes: 0

Facty
Facty

Reputation: 547

#!/bin/bash
# Remove exited containers
/usr/bin/docker ps -a -q -f status=exited | xargs --no-run-if-empty docker rm -v

# Remove dangling images
/usr/bin/docker images -f "dangling=true" -q | xargs --no-run-if-empty docker rmi

# Remove unused images
/usr/bin/docker images | awk '/ago/  { print $3}' | xargs --no-run-if-empty docker rmi

# Remove dangling volumes
/usr/bin/docker volume ls -qf dangling=true | xargs --no-run-if-empty docker volume rm

Run with cron. In my case every hour

enter image description here

Upvotes: 5

Related Questions