Docker overlay2 resource is busy

Question

I need some help with docker! :D

When I have to deploy my application, sometimes I have this error

Cannot create container for service db: error creating overlay mount to /var/lib/docker/overlay2/<HASH>-init/merged: no such file or directory

OR

container <HASH>: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/<HASH>/merged: device or resource busy

This is my docker info output:

Client:
 Debug Mode: false

Server:
 Containers: 17
  Running: 17
  Paused: 0
  Stopped: 0
 Images: 223
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Kernel Version: 3.10.0-1062.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.09GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Live Restore Enabled: false

Everytime that the resource is busy error appears, I move the HASH's directory in /var/lib/docker/(containers OR overlay2) to HASH_old directory and retry to deploy the application, used to work, but now the no such file or directory error always shows up, even after moving all the folders to _old.

Answer 1

When I tried to delete 4 dead docker containers, I got the error

Error response from daemon: container xxxx: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/xxxx/merged: device or resource busy

And I tried to remove the directory manually with

rm -rf /var/lib/docker/overlay2/

I got the error

rm: cannot remove '/var/lib/docker/overlay2/xxxx/merged': Device or resource busy

I check for the mounted filesystem, using

findmnt -l

And found 4 mounted filesystem with source to overlay (due to the 4 containers)

I stopped docker service and disabled docker, then

In order to unmount the containers in the filesystem, I ran:

umount overlay OR umount /var/lib/docker/overlay2/

4 times, in order to remove the 4 mounted instances. (It is umount, not unmount)

And then, I was able to delete the directory

rm -rf /var/lib/docker/overlay2/

without any issue, and also the containers.

I then started my docker service again, It works!

Answer 2

Got this error when cleaning up the overlay2 folder and could solve the problem finally like this:

umount /var/lib/docker/overlay2-legacy/*/merged
rm -R /var/lib/docker/overlay2-legacy/*

Attention: This might lead into data loss.

Answer 3

As JVictorV answer did not work for me, I post here my solution for this general error, many questions are there regards it, but not many solutions or they don't work

In my case, the working workaround surprisingly was to restrict the number of 'RUN' docker building commands/layers, since if the number surpassed 60 layers/commands, it always ended up with that missing 'merged' folder error, no matter what was the contents of the command, even simple command such as RUN ls -la ended up with that error, if the total number of such/any commands was higher than about 60, strange. Merged subfolder was always missing, though even when I automatically generated all the merged subfolders, always was created on the fly a new layer with a new hash, which was missing that subfolder.

Answer 4

Shutdown the buggy containers and run docker system prune -af, that will remove unused containers and images. After that just try to build and deploy, should work