CODEWITHSUNDEEP
linuxdockerspacecentos7
user3139774
user3139774

Reputation: 1335

Docker run error: "Thin Pool has free data blocks which is less than minimum required"

We are trying to run a docker in a way that used to work, but now we get a "Thin Pool lack of space" error:

docker run --privileged -d --net=host --name=fat-redis -v /fat/deploy:/fat/deploy -v /fat/fat-redis/var/log:/var/log -v /home:/home fat-local.indy.xiolab.myserv.com/fat-redis:latest /fat/deploy/docker/fat-redis/fat_start_docker_inner.sh
docker: Error response from daemon: devmapper: Thin Pool has 486 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior.
See 'docker run --help'.

What does this error mean? We tried 'docker rmi' and the advise from here, but all in vain.

Any ideas?

Thank you

Upvotes: 11

Views: 36024

Answers (6)

Lodlaiden
Lodlaiden

Reputation: 391

TL;DR

Sometimes you just need more space. Increase the data file with the truncate command.

Explanation: The reason that a reinstall or a purge of all your images works is that you have a "ramdisk" that docker uses as a space to build the images, but it's not purged after the image is running. If you are running several different images, you can fill up the scratch disk and the "newer" image doesn't have enough space to run in. The docker system prune command won't work because that space is legitimately consumed. You need to increase the size of the scratch file.

  1. Make sure you have extra physical space on disk

    df

  2. Figure out the size of your data file

    docker info |grep 'Data Space'

  3. Find the location of your data file

    docker info |grep 'loop file'

  4. Increase the size of your data file (+50G or whatever)

    sudo truncate -s 150G /var/lib/docker/devicemapper/devicemapper/data

  5. Restart the machine. The guide talks about a bunch of commands to "cascade" the resize through the layer, but a restart did them automatically

    sudo reboot

References:

{all the SO posts that complained about the loopback driver being outdated} https://docs.docker.com/storage/storagedriver/device-mapper-driver/#use-operating-system-utilities

Upvotes: 3

Raj
Raj

Reputation: 93

Use the following to cleanup unnecessary images.

docker image prune -a --force --filter "until=240h"

Refer to this document for more details: https://docs.docker.com/engine/reference/commandline/image_prune/

Upvotes: 2

Jerry Li
Jerry Li

Reputation: 41

You don't have to reinstall Docker. Rather, you can clean up all the containers, images, volumes, etc. under /var/lib/docker directory.

Those images could be pulled up from your Docker repositories again. (This is assuming you only use this Docker host for building Docker images.)

Upvotes: 3

JPS
JPS

Reputation: 21

My issue was unrelated to the loopback device problem, but was generating the same error condition. "docker images -a" showed a number of name=none tag=none images taking up space. These images were not "dangling"; they were referenced by a current active image, and could not be deleted.

My solution was to run "docker save" and write the active image to a tar file, delete the active image (which deleted all the child images), then run "docker load -i" from the tar file and create a single new image. No more errors related to Thin Pool space.

Reinstalling docker would have corrected it, simply because reinstalling docker does clear out all images, but it would have begun building up again and then I would have re-encountered this issue in the future.

Upvotes: 2

aaaarrgh
aaaarrgh

Reputation: 1004

Running with data/metadata over loopback devices was the default on older versions of docker. There are problems with this, and newer versions have changed this default. If docker was configured this way, then normal updates (e.g. through rpm/apt) don't change the configuration, which is why a full reinstall was required to fix.

Here's an article with instructions on how to configure older versions to not use loopback devices: http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/

Upvotes: 9

user3139774
user3139774

Reputation: 1335

Turned out that re-installing docker did the trick. Use the following link: https://docs.docker.com/engine/installation/linux/centos/

Cheers

Upvotes: 0

Related Questions