CentOS 7 : docker in docker: "Error: No space left on device"

Question

I been tackling trying to get docker-in-docker working for a CentOS 7 image, with ubuntu as the host image.

As of now i have not started building this as a docker image. And is currently experimenting with bash on how to "get docker in docker to work"

Currently systemctl start docker ran inside the inner docker image CentOS gives the following error

Error: No space left on device
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.

Further investigation on the error systemctl status docker gives the following

Oct 13 04:32:08 codenvy docker[6520]: time="2015-10-13T04:32:08Z" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
Oct 13 04:32:08 codenvy docker[6520]: time="2015-10-13T04:32:08Z" level=warning msg="Running modprobe bridge nf_nat failed with message: , error: exit status 1"
Oct 13 04:32:08 codenvy docker[6520]: time="2015-10-13T04:32:08Z" level=info msg="-job init_networkdriver() = OK (0)"
Oct 13 04:32:09 codenvy docker[6520]: time="2015-10-13T04:32:09Z" level=warning msg="Your kernel does not support cgroup swap limit."
Oct 13 04:32:09 codenvy docker[6520]: time="2015-10-13T04:32:09Z" level=info msg="Loading containers: start."
Oct 13 04:32:09 codenvy docker[6520]: time="2015-10-13T04:32:09Z" level=info msg="Loading containers: done."
Oct 13 04:32:09 codenvy docker[6520]: time="2015-10-13T04:32:09Z" level=fatal msg="Shutting down daemon due to errors: inotify_add_watch: no space left on device"
Oct 13 04:32:09 codenvy systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 13 04:32:09 codenvy systemd[1]: Failed to start Docker Application Container Engine.
Oct 13 04:32:09 codenvy systemd[1]: Unit docker.service entered failed state.

Additional Information

• Host OS: Ubuntu 14.04.2 LTS
• Docker Image: codenvy/onprem-multi (which is based on centos:centos7)
• Mounted Volumes
  • /sys/fs/cgroup
  • /sys/fs/cgroup:/sys/fs/cgroup:ro
  • /mnt/docker-files-lvm/docker/codenvy/docker:/var/lib/docker
  • /mnt/docker-files-lvm/docker/codenvy/ldap:/var/lib/ldap
  • /mnt/docker-files-lvm/docker/codenvy/mongo:/var/lib/mongo
  • /mnt/docker-files-lvm/docker/codenvy/home:/home
• Privileged mode

Note

This is not regarding how to install codenvy, its regarding getting docker itself installed and working. Before installing codenvy

Added: df -h

Filesystem                               Size  Used Avail Use% Mounted on
/dev/mapper/hc--dawn--vg-root             27G  3.5G   23G  14% /
none                                     4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                      16G   12K   16G   1% /dev
tmpfs                                    3.2G  1.1M  3.2G   1% /run
none                                     5.0M     0  5.0M   0% /run/lock
none                                      16G   37M   16G   1% /run/shm
none                                     100M     0  100M   0% /run/user
/dev/sda1                                236M   95M  129M  43% /boot
/dev/mapper/base--storage-docker--files  886G   52G  790G   7% /mnt/docker-files-lvm

Note: /mnt/docker-files-lvm/docker maps to the /mnt/docker-files-lvm (which is 790GB)

Answer 1

If it is a no space left problem you can configure docker to have it store images and containers elsewhere.

Since you are using systemctl the config file is located here: /lib/systemd/system/docker.service

You can add a -g option to change where docker store things.

For example:

ExecStart=/usr/bin/docker daemon -g /there_is_space_here -H fd://