CODEWITHSUNDEEP
dockerubuntufilesystemslvm
xiaojueguan
xiaojueguan

Reputation: 1050

/var/lib/docker/overlay2/*/merged take too much space than it should be

problem descriptions:

In my system, /var/lib/docker/overlay2/*/merged take too much space than it should be and the detail looks like below.

root@sriov-TaiShan-200:~# df -h --total
Filesystem          Size  Used Avail Use% Mounted on
udev                 63G     0   63G   0% /dev
tmpfs                13G  6.2M   13G   1% /run
/dev/nvme0n1p2      2.9T  2.4T  329G  89% /
tmpfs                63G     0   63G   0% /dev/shm
tmpfs               5.0M     0  5.0M   0% /run/lock
tmpfs                63G     0   63G   0% /sys/fs/cgroup
/dev/loop0          128K  128K     0 100% /snap/bare/5
/dev/loop1           49M   49M     0 100% /snap/core18/2406
/dev/loop2           49M   49M     0 100% /snap/core18/2349
/dev/loop3           58M   58M     0 100% /snap/core20/1437
/dev/loop5          229M  229M     0 100% /snap/gnome-3-38-2004/100
/dev/loop6           73M   73M     0 100% /snap/lxd/23043
/dev/loop7          198M  198M     0 100% /snap/gnome-3-34-1804/73
/dev/nvme0n1p1      512M  5.2M  507M   1% /boot/efi
/dev/loop8           66M   66M     0 100% /snap/gtk-common-themes/1519
/dev/loop9           49M   49M     0 100% /snap/snap-store/545
/dev/loop10         229M  229M     0 100% /snap/gnome-3-38-2004/88
/dev/loop11          39M   39M     0 100% /snap/snapd/15541
/dev/loop14         198M  198M     0 100% /snap/gnome-3-34-1804/75
/dev/loop15          73M   73M     0 100% /snap/lxd/23003
/dev/loop16          53M   53M     0 100% /snap/snap-store/559
//10.67.93.97/work  916G  816G  100G  90% /mnt
/dev/loop17          82M   82M     0 100% /snap/gtk-common-themes/1534
/dev/loop13          39M   39M     0 100% /snap/snapd/15909
/dev/loop12          58M   58M     0 100% /snap/core20/1498
tmpfs                13G   16K   13G   1% /run/user/0
tmpfs               126G   12K  126G   1% /var/lib/kubelet/pods/12baa83c-6696-43f3-b060-85ba620867aa/volumes/kubernetes.io~projected/kube-api-access-j26q8
overlay             2.9T  2.4T  329G  89% /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89/merged
shm                  64M     0   64M   0% /var/lib/docker/containers/9b02ed0d68aa3c24531b1e2852861a202b36b9d7d08645ee170035a3eab48747/mounts/shm
overlay             2.9T  2.4T  329G  89% /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d/merged
total               9.9T  8.0T  1.4T  86% -

root@sriov-TaiShan-200:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          2         2         109.7MB   0B (0%)
Containers      2         2         2.294kB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

root@sriov-TaiShan-200:~# du -h --max-depth=1 /var/lib/docker/overlay2 | sort -rh | head -25
223M    /var/lib/docker/overlay2
111M    /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d
68M /var/lib/docker/overlay2/6b7c00ae09c4df6f35e65114513bd6216c2436e62ae26c3b91b4c2ab60734662
42M /var/lib/docker/overlay2/94b81b09cc4a7ca3ead72888264e3cff25af77d2fc00334847aa2c3bde225e8a
1.6M    /var/lib/docker/overlay2/65ca885064b7df65728ea2864d3f633f035d4b9f3348ae4bcdc8c926157b9552
528K    /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89
488K    /var/lib/docker/overlay2/5e344fbdc03c21013a38b1a82987d7e9094b5d13fe96d0f89f6e31a096f51c9c
48K /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89-init
36K /var/lib/docker/overlay2/l
32K /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d-init

Things i tried:

  1. extend lv /dev/ubuntu-vg/ubuntu-lv size but no lucky
  2. reinstall the docker with below steps but no lucky
apt-get purge docker-ce
rm -rf /var/lib/docker
rm -rf /etc/docker
apt-get remove docker docker-engine docker.io containerd runc
/etc/apt/sources.list.d/docker.list > /dev/null
apt update
mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
apt update
apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
  1. docker system prune -a but no luck
root@sriov-TaiShan-200:~# docker system prune -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

updates

It seems that docker wasn't using that much disk space. Through the du result, it is other things that take the space. However the overlay 2.9T 2.4T 329G 89% /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d/merged is quite confusing.

root@sriov-TaiShan-200:# du -h --max-depth=1 / --exclude=mnt --exclude=proc --exclude=home | sort -rh
2.3T    /
1.6T    /var
686G    /root
15G /usr
6.1G    /snap
3.6G    /x86_64-linux-gnu
1.2G    /boot
88M /opt
16M /etc
6.2M    /run
92K /tmp
16K /lost+found
12K /media
4.0K    /srv
4.0K    /cdrom
0   /sys
0   /dev

Upvotes: 3

Views: 11874

Answers (1)

BMitch
BMitch

Reputation: 264811

The merged folders are overlay filesystems, they take no disk space themselves, instead df reports the disk usage of the underlying filesysem, which in your case is /:

/dev/nvme0n1p2      2.9T  2.4T  329G  89% /
overlay             2.9T  2.4T  329G  89% /var/lib/docker/overlay2/150f4adf713b9882cd39e5580204040f20dcebf20205c9c62b55a85ac416da89/merged
overlay             2.9T  2.4T  329G  89% /var/lib/docker/overlay2/803a832d1a228c7e57b47b9d53244deadc98dbe41248a478d3b6981bc111832d/merged

The disk usage can be anywhere on the root filesystem, so you need to examine the entire drive to find where diskspace is being used. This may have nothing to do with docker.

  1. extend lv /dev/ubuntu-vg/ubuntu-lv size but no lucky

Your root filesystem is listed as /dev/nvme0n1p2, so I'm not sure how changing the lv would affect that.

With containers, common places to use disk are:

  1. image layers
  2. container RW filesystem (from an app writing to the container filesystem)
  3. logs of container output

The quick cleanup in docker is docker system prune. There are more options to that, and things that aren't included in the prune described in this answer.

Upvotes: 0

Related Questions