CODEWITHSUNDEEP
amazon-web-servicesdocker
D'Arcy Rail-Ip
D'Arcy Rail-Ip

Reputation: 11965

No space in docker thin pool

I'm receiving an error trying to launch task definitions in ECS:

CannotPullContainerError: failed to register layer: devmapper: Thin Pool has 4405 free data blocks which is less than minimum required 4480 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

I found this post which has a few recommended steps, but running these does not solve the problem.

Here is the info I receive from docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: docker-202:1-655458-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 45.74 MB
 Data Space Total: 107.4 GB
 Data Space Available: 13.71 GB
 Metadata Space Used: 622.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 4.4.35-33.55.amzn1.x86_64
Operating System: Amazon Linux AMI 2016.09
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.862 GiB
Name: ip-172-31-53-68
ID: W556:CIZO:27KA:JYLI:ZXUS:FTCF:TMU4:5SL5:OD4P:HNP3:PRUM:BUNX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8

I'm really stuck on what to do here... I can't launch any new deploys.

Upvotes: 15

Views: 19685

Answers (3)

Yuri Zinkovets
Yuri Zinkovets

Reputation: 111

This document describes the problem and possible solutions. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/CannotCreateContainerError.html

In my exact case, removal of unused data blocks within containers helped: On the BS instance: sudo sh -c "docker ps -q | xargs docker inspect --format='{{ .State.Pid }}' | xargs -IZ fstrim /proc/Z/root/"

Upvotes: 3

iGili
iGili

Reputation: 883

Had the same issue. The solution in the post you've mentioned does not remove unused images.

$ Docker system prune -a did the trick.

More details here

Upvotes: 8

Sunny
Sunny

Reputation: 425

Try this :-

# docker volume rm $(docker volume ls -qf dangling=true)
# docker rm $(docker ps -q -f 'status=exited')

Upvotes: 3

Related Questions