Docker overlay2: error walking file system: OSError [Errno 40] Too many levels of symbolic links

Question

Main app: uvicorn server on starlette (python) webapp

While I was trying to debug the error in the title (troubleshoot log is following below) running the below command at the host's FS (/var/lib/docker/overlay2/[IMAGE_HASH_FOLDER]

find -L ./ -mindepth 15

I find the files involved in the loop. Locally is the /usr/bin/X11 and at server I'm getting the following:

error walking file system: OSError [Errno 40] Too many levels of symbolic links: '/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/fd/6/dev/stderr'

The owner of the conflicting files (FS is the one of the host), pending docker service restart after the pruning:

➜  overlay2 find -L ./ -mindepth 15          

find: File system loop detected; ‘./d2dba43e7cdbdec81bac529bb85908a7f859f227cda0149389f164272cb372e8/diff/usr/bin/X11’ is part of the same file system loop as ‘./d2dba43e7cdbdec81bac529bb85908a7f859f227cda0149389f164272cb372e8/diff/usr/bin’.

   find: File system loop detected; ‘./6ec18b03535c1dac329e05b2abdc68fb0eea742a06878d90f84c4de73ea6a4a9/merged/usr/bin/X11’ is part of the same file system loop as ‘./6ec18b03535c1dac329e05b2abdc68fb0eea742a06878d90f84c4de73ea6a4a9/merged/usr/bin’.

     find: File system loop detected; ‘./l/GCDLBXTJXAL5PFTI4BE3MM3OE2/usr/bin/X11’ is part of the same file system loop as ‘./l/GCDLBXTJXAL5PFTI4BE3MM3OE2/usr/bin’.
    
    ➜  overlay2 ls -l ./d2dba43e7cdbdec81bac529bb85908a7f859f227cda0149389f164272cb372e8/diff/usr/bin/X11
    lrwxrwxrwx 1 root root 1 May  3  2017 ./d2dba43e7cdbdec81bac529bb85908a7f859f227cda0149389f164272cb372e8/diff/usr/bin/X11 -> .

The dockerfile:

FROM python:3.8

COPY src/ ./

RUN /usr/local/bin/python -m pip install --upgrade pip || true
RUN pip install -r requirements.txt || true

ARG POSTGRES_USER
ENV POSTGRES_USER=$POSTGRES_USER
ARG POSTGRES_PASSWORD
ENV POSTGRES_PASSWORD=$POSTGRES_PASSWORD
ARG POSTGRES_SERVER
ENV POSTGRES_SERVER=$POSTGRES_SERVER
ARG POSTGRES_DB
ENV POSTGRES_DB=$POSTGRES_DB
ARG POSTGRES_PORT
ENV POSTGRES_PORT=$POSTGRES_PORT
ARG SESSION_SECRET
ENV SESSION_SECRET=$SESSION_SECRET
ARG DO_YOU_WANT_USERS
ENV DO_YOU_WANT_USERS=$DO_YOU_WANT_USERS
ARG WHERE_AM_I
ENV WHERE_AM_I=$WHERE_AM_I
# SSL
ARG FORWARDED_ALLOW_IPS
ENV FORWARDED_ALLOW_IPS=$FORWARDED_ALLOW_IPS
ARG SSL_CERTIFICATE
ENV SSL_CERTIFICATE=$SSL_CERTIFICATE
ARG SSL_KEYFILE
ENV SSL_KEYFILE=$SSL_KEYFILE
ARG UPLOADS_PATH
ENV UPLOADS_PATH=$UPLOADS_PATH

RUN echo "FINAL STAGE - RUN APP"

EXPOSE 7000
CMD ["python", "run.py"]

Either I run the container with the volume I usually bind:

UPLOADS_PATH=/var/opt/tmp
LOCAL_UPLOADS_PATH=/var/containers/TEST_UPLOADS

docker build --build-arg POSTGRES_USER --build-arg POSTGRES_PASSWORD --build-arg POSTGRES_SERVER --build-arg POSTGRES_DB --build-arg POSTGRES_PORT --build-arg UPLOADS_PATH --build-arg WHERE_AM_I --build-arg SESSION_SECRET --build-arg DO_YOU_WANT_USERS -t test .

docker run -d --name test_container -v ${LOCAL_UPLOADS_PATH}:${UPLOADS_PATH} -p 7000:7000 test

or without the binding, I still get the same error logs & the app is constantly restarting after every request.

How is possible to have such a loop (linked files?) inside the image?

UPDATE

The container was running smoothly until I've changed bcrypt library with pybcrypt and uvicorn with its cythonized version.

Much appreciate any suggestions on what to further explore.

P.S. I've also tried the docker system prune -a, and although there were deprecated stuff, nothing changed.

P. S. 2: @jordanvrtanoski I've separated the question as you've suggested.

UPDATE #2

Following @jordanvrtanoski inspect command:

➜ docker image inspect -f $'{{.RepoTags}}\t{{.GraphDriver.Data.LowerDir}}' $(docker images -q)

[test:latest]   /var/lib/docker/overlay2/99e3b5db623ae543d045cc86c2d7d36400c8d1780ec4b86c297f5055bbdfe81a/diff:/var/lib/docker/overlay2/4ed6de1627ba5957c8fa9834c797a60d277c76e61f138d1b6909c55ef5475523/diff:/var/lib/docker/overlay2/7f790257bc4e6ed9e6ea6ef5bed0eb0cf3af213ea913484a40946a45639d8188/diff:/var/lib/docker/overlay2/c8e04185bdc7714e116615a3599a9832ebe2080b43f09b68331cca5d7c109371/diff:/var/lib/docker/overlay2/9ef94affd46bbcc11d62999ab0c59d6bf28cc6d51f13a7513b93bb209738940a/diff:/var/lib/docker/overlay2/62438cdccba1f312f34e8458e4ec695019e6af65107b2e16c3d7eaa53ca03c06/diff:/var/lib/docker/overlay2/9ec57b8b2680944690cdceae73c1c49b31716bd5efbed78bd3d54810bffdc7b6/diff:/var/lib/docker/overlay2/b2c4ce8d2b6764476a452489f58e615fcce939eaecb3d65466f81f5f115a5b5d/diff:/var/lib/docker/overlay2/f8609908601489fb7e3e28a32c423ee556ec041c69ba274a02de316ccbef5c48/diff:/var/lib/docker/overlay2/dcd13187b642277de35f299c1abb1d7d9695972e8b8893267a62f65338679080/diff:/var/lib/docker/overlay2/e2ed1696e3a34e69ed493da3a2c10b942f09384b1cebac54afebea6fef9c4521/diff
[python:3.8]    /var/lib/docker/overlay2/c8e04185bdc7714e116615a3599a9832ebe2080b43f09b68331cca5d7c109371/diff:/var/lib/docker/overlay2/9ef94affd46bbcc11d62999ab0c59d6bf28cc6d51f13a7513b93bb209738940a/diff:/var/lib/docker/overlay2/62438cdccba1f312f34e8458e4ec695019e6af65107b2e16c3d7eaa53ca03c06/diff:/var/lib/docker/overlay2/9ec57b8b2680944690cdceae73c1c49b31716bd5efbed78bd3d54810bffdc7b6/diff:/var/lib/docker/overlay2/b2c4ce8d2b6764476a452489f58e615fcce939eaecb3d65466f81f5f115a5b5d/diff:/var/lib/docker/overlay2/f8609908601489fb7e3e28a32c423ee556ec041c69ba274a02de316ccbef5c48/diff:/var/lib/docker/overlay2/dcd13187b642277de35f299c1abb1d7d9695972e8b8893267a62f65338679080/diff:/var/lib/docker/overlay2/e2ed1696e3a34e69ed493da3a2c10b942f09384b1cebac54afebea6fef9c4521/diff

UPDATE #3

So after following both @jordanvrtanoski advices & this post (@Janith Shanilka): Docker overlay2 eating Disk Space

I was missing the following file:

nano /etc/docker/daemon.json

and populated with:

{
  "storage-driver": "aufs"
}

then sudo systemctl restart docker

Now the app doesn't crash, but I'm still getting at logs the same loop message:

error walking file system: OSError [Errno 40] Too many levels of symbolic links: '/usr/bin/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/cc'

also @jordanvrtanoski

➜  docker image inspect -f $'{{.RepoTags}}\t{{.GraphDriver.Data.LowerDir}}' $(docker images -q)


[test:latest]   <no value>
[python:3.8]    <no value>

I've also noticed that df is a little weird, it looks like the docker's volume is like a 'clone' of the basic host's filesystem?

➜  df
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             4046520        0   4046520   0% /dev
tmpfs             815676     3276    812400   1% /run
/dev/sda3       49014600 20123088  26798560  43% /
tmpfs            4078368      304   4078064   1% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
tmpfs            4078368        0   4078368   0% /sys/fs/cgroup
/dev/sda1         474730   148714    296986  34% /boot
tmpfs             815672        0    815672   0% /run/user/0
none            49014600 20123088  26798560  43% /var/lib/docker/aufs/mnt/0d98503bd3ea82e353f6776c2d813a642536ad6dd4300299a8fc22b5d6348bc8

UPDATE #4

So after @jordanvrtanoski 's suggestion I returned docker to overlay, from 'aufs'. The below results are from the host:

➜ cd /var/lib/docker/overlay2 ➜ find -L ./ -mindepth 15 find: File system loop detected; ‘./2ecf467259235c8c4605b058bff4f80100790ee7f5010d4954d6aab1a7f28686/merged/usr/bin/X11’ is part of the same file system loop as ‘./2ecf467259235c8c4605b058bff4f80100790ee7f5010d4954d6aab1a7f28686/merged/usr/bin’. find: File system loop detected; ‘./6f39e8e2089c99f636da9a534e2ccbe7e41202eeb2ce645efa9387dd0ef0b908/diff/usr/bin/X11’ is part of the same file system loop as ‘./6f39e8e2089c99f636da9a534e2ccbe7e41202eeb2ce645efa9387dd0ef0b908/diff/usr/bin’. find: File system loop detected; ‘./l/5AOADDMRCAKLG2FQDDJEYC6CY2/usr/bin/X11’ is part of the same file system loop as ‘./l/5AOADDMRCAKLG2FQDDJEYC6CY2/usr/bin’.

Answer 1

In case someone else stumbles across this later - it looks like uvicorn accesses all subfolders of the path you invoke it from. If you're not explicitly setting a working directory in your dockerfile/compose.yaml, this will be the file system root, which gets into all the bind mount infrastructure in proc that you probably don't care about for running an ASGI server.

WORKDIR /home in a dockerfile or working_dir: /home in a compose.yaml should generally be a fine workaround for most Docker use cases for this error, or to your app directory if you're volume mounting in code.

Answer 2

UPDATE #5

Found the cause: lib uvicorn[standard] is the cythonized version of itself. Once I removed it all errors were gone. So I'll move this to uvicorn's github.

@jordanvrtanoski Thank you once again for your help!

Answer 3

This problem is caused by an self-referencing symbolic linl in the pyhton:3.8 image.

~# docker run -ti --rm python:3.8 bash

root@ef6c6f4e18ff:/# ls -l /usr/bin/X11/X11
lrwxrwxrwx 1 root root 1 May  3  2017 /usr/bin/X11/X11 -> .

The fix the circular reference is caused by python:3.8 image you can simply delete the /usr/bin/X11/X11 symbolic link

root@ef6c6f4e18ff:/# rm /usr/bin/X11/X11

You can add this to your build file as follwos:

FROM python:3.8

COPY src/ ./

RUN rm /usr/bin/X11/X11

RUN /usr/local/bin/python -m pip install --upgrade pip || true
RUN pip install -r requirements.txt || true

ARG POSTGRES_USER
ENV POSTGRES_USER=$POSTGRES_USER
ARG POSTGRES_PASSWORD
ENV POSTGRES_PASSWORD=$POSTGRES_PASSWORD
ARG POSTGRES_SERVER
ENV POSTGRES_SERVER=$POSTGRES_SERVER
ARG POSTGRES_DB
ENV POSTGRES_DB=$POSTGRES_DB
ARG POSTGRES_PORT
ENV POSTGRES_PORT=$POSTGRES_PORT
ARG SESSION_SECRET
ENV SESSION_SECRET=$SESSION_SECRET
ARG DO_YOU_WANT_USERS
ENV DO_YOU_WANT_USERS=$DO_YOU_WANT_USERS
ARG WHERE_AM_I
ENV WHERE_AM_I=$WHERE_AM_I
# SSL
ARG FORWARDED_ALLOW_IPS
ENV FORWARDED_ALLOW_IPS=$FORWARDED_ALLOW_IPS
ARG SSL_CERTIFICATE
ENV SSL_CERTIFICATE=$SSL_CERTIFICATE
ARG SSL_KEYFILE
ENV SSL_KEYFILE=$SSL_KEYFILE
ARG UPLOADS_PATH
ENV UPLOADS_PATH=$UPLOADS_PATH

RUN echo "FINAL STAGE - RUN APP"

EXPOSE 7000
CMD ["python", "run.py"]