CODEWITHSUNDEEP
linuxdocker
Vikram
Vikram

Reputation: 643

Docker Devmapper space issue - increase size

I have the same issue as in space issue on docker devmapper and CentOS7

It only specifies to clean up but not how I can increase the space and I dont have any images to clean. I tried several things with dm.min_free_space but nothing worked and want to increase the space.

OS Version/build: Red Hat Enterprise Linux Server release 7.3 (Maipo)

App version:

Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-11.el7.centos.x86_64
 Go version:      go1.7.4
 Git commit:      96d83a5/1.12.6
 Built:           Tue Mar  7 09:23:34 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-11.el7.centos.x86_64
 Go version:      go1.7.4
 Git commit:      96d83a5/1.12.6
 Built:           Tue Mar  7 09:23:34 2017
 OS/Arch:         linux/amd64

Steps to reproduce

I have no containers running currently and have some docker images pertaining to Kubernetes which will be used by the Kubernetes service.

sudo docker ps -a
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    [kubeuser4@kubenode4 Employee]$ sudo docker images
    REPOSITORY                                            TAG                 IMAGE ID            CREATED             SIZE
    docker.io/busybox                                     latest              00f017a8c2a6        5 days ago          1.11 MB
    registry.access.redhat.com/rhel7/pod-infrastructure   latest              34d3450d733b        6 weeks ago         205 MB
    docker.io/java                                        8                   d23bdf5b1b1b        8 weeks ago         643.1 MB
    gcr.io/google_containers/heapster_grafana             v2.6.0-2            b43443930626        12 months ago       230 MB

When I try to create a docker image of my application that needs to be used, I get the below error.

devmapper: Thin Pool has 8783 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

I tried the cleaning up as mentioned in the other forums, but not helped much and getting the same error. When I tried to run with this sudo docker --storage-opt dm.min_free_space=0%, seems like it starts as a daemon, but still it failed with another error "docker-runc not installed on system" and also I dont want to run it as a daemon.

Below are some command outputs

sudo dmsetup status
localvg00-lv_home: 0 20971520 linear
localvg00-lv_home: 20971520 20971520 linear
docker-251:5-134039-pool: 0 209715200 thin-pool 924 848/524288 1629226/1638400 - rw discard_passdown queue_if_no_space
localvg00-lv_tmp: 0 4194304 linear
localvg00-lv_swap: 0 8388608 linear
localvg00-lv_root: 0 2097152 linear
localvg00-lv_root: 2097152 20971520 linear
localvg00-lv_usr: 0 16777216 linear
localvg00-lv_var: 0 8388608 linear
localvg00-lv_var: 8388608 62914560 linear

sudo docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: docker-251:5-134039-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 106.8 GB
 Data Space Total: 107.4 GB
 Data Space Available: 601.2 MB
 Metadata Space Used: 3.473 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: overlay null bridge host
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 4.1.12-61.1.28.el7uek.x86_64
Operating System: Oracle Linux Server 7.3
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 7.545 GiB
Name: kubenode4

I had also tried increasing all the physical volume size and logical volume size(lv_var) on my linux machine, but still it doesnt work. 

sudo lvs
[sudo] password for kubeuser4:
  LV      VG        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home localvg00 -wi-ao---- 20.00g
  lv_root localvg00 -wi-ao---- 11.00g
  lv_swap localvg00 -wi-ao----  4.00g
  lv_tmp  localvg00 -wi-ao----  2.00g
  lv_usr  localvg00 -wi-ao----  8.00g
  lv_var  localvg00 -wi-ao---- 34.00g


 sudo ls -lsh /var/lib/docker/devicemapper/devicemapper/data
2.3G -rw------- 1 root root 100G Mar 14 22:16 /var/lib/docker/devicemapper/devicemapper/data

Someone please let me know how it can be done.

Thanks,

Upvotes: 3

Views: 4809

Answers (2)

gaoithe
gaoithe

Reputation: 4368

It is better move away from devicemapper for a few reasons.

devicemapper in loopback unrecoverable storage issue: https://github.com/docker/docker/issues/3182 "devicemapper not recommended for production use".

I found it easy enough to switch to overlay storage driver, YMMV of course but hopefully not too much. 'rm -rf /var/lib/docker' is somewhat optional when switching but easy and I would highly recommend it as long as you can load your images back in. http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/

systemctl stop docker
rm -rf /var/lib/docker

# if these files do not already exist . . . create them, otherwise you need to edit by hand, you can also just add -s overlay in the systemctl docker script
ls /etc/sysconfig/docker /etc/sysconfig/docker-storage
[[ $? != 0 ]] && {
    echo OPTIONS='--selinux-enabled=false' > /etc/sysconfig/docker
    echo "DOCKER_STORAGE_OPTIONS= -s overlay" > /etc/sysconfig/docker-storage
}
systemctl start docker
systemctl status docker
docker images

more reading: https://docs.docker.com/engine/userguide/storagedriver/selectadriver/ https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/

Upvotes: 2

Vikram
Vikram

Reputation: 643

Was able to get it working and have mentioned it in

https://forums.docker.com/t/devmapper-space-issue/29786/3

Upvotes: 1

Related Questions