Arch Linux, Docker "No space left on device."

Question

All of the similar questions I see are resolved by cleaning up the images or containers or orphaned volumes but I am not having any of those problems. I even completely deleted /var/lib/docker and still nothing.

Relevant output:

[N] ⋊> ~/W/W/cocagne on master ⨯ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v /var/lib/docker:/var/lib/docker martin/docker-cleanup-vol
umes
docker: Error response from daemon: Container command '/usr/local/bin/docker-cleanup-volumes.sh' not found or does not exist..
[N] ⋊> ~/W/W/cocagne on master ⨯ docker-compose build                                                                                                11:56:23
mysql uses an image, skipping
Building vitess
Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 9, in <module>
    load_entry_point('docker-compose==1.7.1', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 58, in main
    command()
  File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 109, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 213, in build
    force_rm=bool(options.get('--force-rm', False)))
  File "/usr/lib/python3.5/site-packages/compose/project.py", line 300, in build
    service.build(no_cache, pull, force_rm)
  File "/usr/lib/python3.5/site-packages/compose/service.py", line 718, in build
    buildargs=build_opts.get('args', None),
  File "/usr/lib/python3.5/site-packages/docker/api/build.py", line 54, in build
    path, exclude=exclude, dockerfile=dockerfile, gzip=gzip
  File "/usr/lib/python3.5/site-packages/docker/utils/utils.py", line 103, in tar
    t.add(os.path.join(root, path), arcname=path, recursive=False)
  File "/usr/lib/python3.5/tarfile.py", line 1938, in add
    self.addfile(tarinfo, f)
  File "/usr/lib/python3.5/tarfile.py", line 1966, in addfile
    copyfileobj(fileobj, self.fileobj, tarinfo.size)
  File "/usr/lib/python3.5/tarfile.py", line 244, in copyfileobj
    dst.write(buf)
  File "/usr/lib/python3.5/tempfile.py", line 483, in func_wrapper
    return func(*args, **kwargs)
OSError: [Errno 28] No space left on device
[I] ⋊> ~/W/W/cocagne on master ⨯ docker ps -a                                                                                                        11:56:30
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[I] ⋊> ~/W/W/cocagne on master ⨯ docker ps -q                                                                                                        11:57:25
[I] ⋊> ~/W/W/cocagne on master ⨯ docker image -q                                                                                                     11:57:28
docker: 'image' is not a docker command.
See 'docker --help'.
[I] ⋊> ~/W/W/cocagne on master ⨯ docker images -a                                                                                                    11:57:39
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
martin/docker-cleanup-volumes   latest              8c41df286c03        12 weeks ago        22.12 MB
[I] ⋊> ~/W/W/cocagne on master ⨯ df -h                                                                                                               11:57:41
Filesystem      Size  Used Avail Use% Mounted on
dev             3.9G     0  3.9G   0% /dev
run             3.9G  832K  3.9G   1% /run
/dev/sda4        27G  9.1G   17G  36% /
tmpfs           3.9G   64M  3.8G   2% /dev/shm
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G   32K  3.9G   1% /tmp
/dev/sda1        42G   16G   25G  39% /home
/dev/sda2        42G  9.4G   30G  24% /var
/dev/sda5       1.3G   32M  1.3G   3% /boot
tmpfs           790M   12K  790M   1% /run/user/1000
[I] ⋊> ~/W/W/cocagne on master ⨯                                                                                                                     11:57:54

docker info

[I] ⋊> ~/W/W/cocagne on master ⨯ docker info                                                                                                         12:01:55
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: docker-8:2-2359321-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.8 MB
 Data Space Total: 107.4 GB
 Data Space Available: 34.57 GB
 Metadata Space Used: 581.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.131 (2016-07-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.6.4-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.706 GiB
Name: crockford
ID: HO2U:ELWR:LDB3:PMEY:5YOJ:D7YJ:2HJA:PVYG:45K2:J6KI:D6WO:4RUE
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

One thing that makes my issue a little different (Where I think the root of the issue comes from)

Before I created a separate partition for /var, it was on my root partition, which eventually maxed out. Once it maxed out, I shrunk my home partition, create a /var partition, copied my root's /var to my new /var, and removed my old /var. But for some reason, docker still think's it's maxed out? I have no idea.

I also tried to resinstall docker with sudo pacman -S docker but nothing.

Edit: I just tried it with a normal docker build . and that works fine. Somehow docker-compose thinks it's out of memory though?

Answer 1

Try this:

mount -o remount,size=4G,noatime /tmp

Answer 2

The python stack trace from docker-compose indicates that it can't seem to create a temporary file. This would indicate there's no space left in /tmp.

OP mentioned that his RAM is completely consumed when he runs docker-compose in the comments. Given that and the fact that /tmp is mounted on tmpfs it makes sense that there is no space left for Python/docker-compose to create any temporary files in /tmp.

The possible solutions are:

• Temporarily switch the default tempfile generation location by setting one of the following environment variables: TMPDIR, TEMP, TMP (ref: Python doc)
• Change /tmp to not use tmpfs and use disk instead.
• Increase the amount of RAM/Swap space on your machine. (You can increase swap without messing with your partitions like so). tmpfs is backed by volatile storage, which means both RAM and Swap should theoretically work.

Note, most of these cases will result in a slowdown of your application, especially if the docker build process is I/O heavy.