CODEWITHSUNDEEP
dockerdocker-composedocker-machine
Marc Perrin-Pelletier
Marc Perrin-Pelletier

Reputation: 13456

Docker Machine: No space left on device

I'm trying to set up Docker Machine with Docker Compose.

Scenario 1 (without Docker Machine)
If I run docker-compose up -d without Docker Machine, it creates my 3 linked containers as intented (nginx+mongodb+nodejs).

Scenario 2 (with Docker Machine)
Then I create a VM using Docker Machine and tell Docker to talk to that machine with eval $(docker-machine env streambacker-dev).

At this point, if I ssh to my docker machine and run df -h, I get:

docker machine df -h

If I then run docker-compose up -d, I get a "no space left on device" error while downloading the last container.

"tmpfs" seems to be indeed a bit full after that:

docker machine df -h

Checking the --virtualbox-disk-size option shows that it defaults to 20000 MB, which I think is what we can see as "/dev/sda1" on both pictures. So why are containers filling up "tmpfs" n and what exactly is "tmpfs"? Is is a temporary download directory? How can I create more space for my containers?

Thanks!

For information, I'm using Docker Machine 0.4.0-rc2 and Docker Compose 1.3.2.

Upvotes: 152

Views: 128289

Answers (9)

jbyler
jbyler

Reputation: 7847

In addition to images, container log files can consume a lot of disk space, and there is no easy way to inspect them, especially on macOS (see this answer for the only way I have found).

The simplest way to reclaim the disk space used by a container is to docker rm the container and recreate it.

You can also find the path to the log file with docker inspect --format='{{.LogPath}}' CONTAINER_NAME_HERE and remove it from either your host machine's file system (if the host OS is Linux) or from the Docker Machine VM's file system (if the host OS is macOS). More details are on my answer to a related question.

Upvotes: 0

kiriloff
kiriloff

Reputation: 26335

Open file /lib/systemd/system/docker.service with your favorite text editor and replace the following line where /new/path/docker is a location of your new chosen docker directory:

FROM:
ExecStart=/usr/bin/docker daemon -H fd://
TO:
ExecStart=/usr/bin/docker daemon -g /new/path/docker -H fd://

When ready stop docker service:

# systemctl stop docker

check that docker service is stopped, the following command should yield no result

# ps aux | grep -i docker | grep -v grep

reload daemon

# systemctl daemon-reload

Once this is done create a new directory you specified above and optionally rsync current docker data to a new directory:

# mkdir /new/path/docker
# rsync -aqxP /var/lib/docker/ /new/path/docker

# systemctl start docker

confirms that docker runs into new directory

#  ps aux | grep -i docker | grep -v grep

https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux

Upvotes: 0

Alex Efimov
Alex Efimov

Reputation: 3703

If you are using Docker Community Edition:

 docker system prune --volumes

If you are using boot2docker (docker-machine) clear the volumes that are orphaned:

 docker volume rm $(docker volume ls -qf dangling=true)

Clear unused images:

 docker rmi $(docker images -q -f "dangling=true")

Upvotes: 95

kwabena Adu-Darkwa
kwabena Adu-Darkwa

Reputation: 41

Run docker system df to see what is taking up space on your docker

Then run:

docker system prune --all --force

to remove all hidden and unused containers.

docker system prune does not remove all unused containers.

Upvotes: 4

aabiro
aabiro

Reputation: 4238

I ran into this problem and could not add additional space with the docker UI for mac, I installed docker with homebrew and ran the following command when creating my machine:

docker-machine create --driver virtualbox --virtualbox-memory "2048" --virtualbox-disk-size "40000" default

this adds double the space for memory and disk size to the virtualbox that I had before and you can add the settings size here that you need as you see fit

Upvotes: 0

Mahmoud Zalt
Mahmoud Zalt

Reputation: 31090

I had the same error ([ERROR] InnoDB: Error number 28 means 'No space left on device') and solve it this way:

1 . Delete the orphaned volumes in Docker, you can use the built-in docker volume command. The built-in command also deletes any directory in /var/lib/docker/volumes that is not a volume so make sure you didn't put anything in there you want to save.

Warning be very careful with this if you have some data you want to keep

Cleanup:

$ docker volume rm $(docker volume ls -qf dangling=true)

Additional commands:

List dangling volumes:

$ docker volume ls -qf dangling=true

List all volumes:

$ docker volume ls

2 . Also consider removing all the unused Images.

First get rid of the <none> images (those are sometimes generated while building an image and if for any reason the image building was interrupted, they stay there).

here's a nice script I use to remove them 

docker rmi $(docker images | grep "^<none>" | awk '{print $3}')

Then if you are using Docker Compose to build Images locally for every project. You will end up with a lot of images usually named like your folder (example if your project folder named Hello, you will find images name Hello_blablabla). so also consider removing all these images

you can edit the above script to remove them or remove them manually with

docker rmi {image-name}

Upvotes: 167

johndpope
johndpope

Reputation: 5249

On docker osx / I was able to press a button [Move Disk Image] and it successfully moved the Docker.qcow2 (presumably containing containers / images)

enter image description here initially - when machines started - I was still getting a No space left on device error but it resolved shortly after.

Upvotes: 2

andhdo
andhdo

Reputation: 963

A. REMOVE UNUSED IMAGES

using the docker rm or docker rmi commands you can remove the images that you don't need. Actually exist an image that helps in this task (martin/docker-cleanup-volumes). The basis is to start selectig from your images and containers list:

docker ps -a -s

B. MODIFY THE DOCKER JSON DESCRIPTOR

it's mentioned in some forums. The idea is to increment the descriptor located in ~/.docker/machine/machines/default/config.json . The param seems to be DiskSize but i don't know if it works in other OSs (not in windows).

C. LINUX RESIZE:

in Windows OS, docker machine or boot2docker is in fact a virtualbox vm, then you can follow the procedure to resize the disk. Take care to backup the files. The general procedure is to make a resize in virtualbox and then use an utilitary called gpartd to modify the space perceived by linux in its partitions. There are some links to do this procedure referenced below:

D. RECREATE THE DOCKER-MACHINE / BOOT2DOCKER

The idea is recreate the default docker-machine. The following commands can illustrate you. Note that as you are re-creating the boot2docker, you will lost the previous downloaded docker images.

docker-machine rm default

docker-machine create --driver virtualbox --virtualbox-disk-size "100100" default

docker-machine env default

then you can go to virtual box and see the boot2docker space with the command "df -h"

Upvotes: 15

wtekiela
wtekiela

Reputation: 1969

Like said above, the tmpfs has nothing to do with --virtualbox-disk-size. It seems like boot2docker mounts tmpfs into memory, so you need to dedicate more memory to your virtualbox vm. You can do it by specifying the --virtualbox-memory parameter.

   --virtualbox-memory "1024"
Size of memory for host in MB [$VIRTUALBOX_MEMORY_SIZE]

Defaults:

$ docker-machine create --driver virtualbox testA
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
$ docker-machine ssh testA
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@testA:~$ df -h /
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   896.6M    112.7M    783.9M  13% /

With --virtualbox-memory set to 8096

$ docker-machine create --driver virtualbox --virtualbox-memory 8096 testB
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
$ docker-machine ssh testB
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@testB:~$ df -h /
Filesystem                Size      Used Available Use% Mounted on
tmpfs                     6.9G    112.4M      6.8G   2% /

Upvotes: 90

Related Questions