dockerd error Upload failed: file integrity checksum failed for \"var/log/alternatives.log\"

Question

We have our own docker registry for storing our images. In this registry we can push images without issues except for one project using nginx.

This is the error we're experiencing on pushing the image, building process is fine:

Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\"

We've tried even removing that file in our Docker file when building and pushing the image without success.

Does anybody know what's happening here and how to fix that? Is there any way to avoid this kind of integrity checksum validations maybe? or even better, does anybody know how to not perform integrity checksum validations for log files?

Thanks!

Here you can find the complete output of docker daemon running on debug mode:

dockerd[37603]: level=debug msg="Calling GET /_ping"
dockerd[37603]: level=debug msg="Calling POST /v1.29/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=docker-registry.cobrowser.io%2Fnginx-edge%3Adevelop_8d07da6&target=&ulimits=null"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c apt-get clean all && apt-get update && apt-get upgrade -y --no-install-recommends   && apt-get install -y --no-install-recommends dnsutils]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c rm /etc/nginx/conf.d/*.conf]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop) WORKDIR /usr/src]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop) COPY file:d9f89463e547e958c06f20b31d4e01baade81f889c0cf4c2c07d71f6ef9a20b7 in /usr/src/ ]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c chmod +x /usr/src/start.sh]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop) COPY file:a623599d27eb98683ad3e430f3039e039687aeaf4aa0772b82ce1d34a05f020d in /etc/nginx/ ]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop) COPY multi:66f62c9f775297ff6eb89ed59805aa6396933472251ab3c88ce9fb09bf223626 in /etc/nginx/conf.d/ ]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop) COPY file:a141aa786391be73d67da8e1b24826f0bf8b286c9c34f01e95e71a18fafe3a7c in /etc/nginx/ ]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop)  ENV DOMAIN_NAME=local.cobrowser.io]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop)  ENV API_SERVICE_HOST=127.0.0.1]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop)  ENV API_SERVICE_PORT=80]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c mkdir -p /var/www]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c chown www-data:www-data /var/www]"
dockerd[37603]: level=debug msg="[BUILDER] Use cached version: [/bin/sh -c #(nop) COPY dir:9881d98ba2bede4742ef22ac79ee1761eeca09a85be229bfee8ca7daad1e5d21 in /var/www ]"
dockerd[37603]: level=debug msg="Calling GET /_ping"
dockerd[37603]: level=debug msg="Calling POST /v1.29/images/docker-registry.cobrowser.io/nginx-edge/push?tag=develop_8d07da6"
dockerd[37603]: level=debug msg="hostDir: /etc/docker/certs.d/docker-registry.cobrowser.io"
dockerd[37603]: level=debug msg="crt: /etc/docker/certs.d/docker-registry.cobrowser.io/ca.crt"
dockerd[37603]: level=debug msg="hostDir: /etc/docker/certs.d/docker-registry.cobrowser.io"
dockerd[37603]: level=debug msg="crt: /etc/docker/certs.d/docker-registry.cobrowser.io/ca.crt"
dockerd[37603]: level=debug msg="Trying to push docker-registry.cobrowser.io/nginx-edge to https://docker-registry.cobrowser.io v2"
dockerd[37603]: level=debug msg="Pushing repository: docker-registry.cobrowser.io/nginx-edge:develop_8d07da6"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:9a6636881be53cb07b2d53161217671f21465d5954725bb31d80a015e8542575 (sha256:5954625e632c0d43fcd4f5065d6649905799b06966c1c5de4014817a364f4e5f) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:be716701512e7ca80b653db830299f35159e957e75294684147ff705a2f89381 (sha256:76b276f8dc85ca891b9ccb153bbcad3580ff8d54ac5707bc4779e34a4c4ee522) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:d8526380557eff001a798e2ab40b0f4e6f4bee9a3556cd9454b019fe731b5cb8 (sha256:15ad477e73388863ea6a675b25af6b64b26a93c7a8795e639b94ec457ad84049) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:21ad1901a4a9ed8251e271dcc76a7764dbb54f6a03cf35af300cad2e7edcb616 (sha256:9fcd78b0bcffb236af5115eeeb3550c29719f5abdc288a92c1bfa6a8e08c083c) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:8deca65b0d19c8684ca09ffa7b0a7c8356dfff53f092c100120a2173a78a5e12 (sha256:339526da58af0e24b1c7bdbcaeb5525a9e0e2927858fab803433f900f4b1c8b3) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:fc106b5a26a13a8c67174e5395ea9af0dd820a121a441f653f4d349c4bfe2248 (sha256:dc0bc2faa68b33a92d852b8f6eec33f7ea19e34803e048144ff70a74db76761f) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:a6c57793448c52bb04264e000e38a65524f6330a34f675df807ca9de5808ec67 (sha256:375731d292cc307d2f04d3a4e54300cd1c3d7d27d1f4b675a317543666d7d5e6) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:a55d03d4a181996bf653167f6d57924fc16fcf781f510b93f6382349e66fc6af (sha256:238cf06b9cd8d2aaca4f1212b6e27d175847f5bfa36a590e0ae9186fe08bb7df) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:02940c312545bc1f805c1bdde1c6fed165d79b2d98b187814d1f5d336c1115c9 (sha256:e3a4c6a3bda5cf475fd44dddbceb2661eca162bc269bd7d5c933ca2c8fed7a20) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:8a30b8e3bcc4b586f687be40466bafd97b97db1f918a0844eb6dd91a2282fd46 (sha256:9ec4a507400020e9a0ea407d07807d128e90658d37afbf821ce19418d6554052) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:a82b6c66a6d41a9d1eb8a9ce90ade6c7fd61c98cd6e764a79c9812ed4327f08c (sha256:06b6780060bd47da88e342e66d34ec57f76f3fb290659beda1e5c493ebed3895) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:1941ca4a7a8446a0dd9d1572898c1d09e590b2de313880e41cc00cf02cd191ea (sha256:31123d939af127bfa829e86ce412ca1a010c56b2db2b8758da6f4e87e30a5377) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Checking for presence of layer sha256:a2ae92ffcd29f7ededa0320f4a4fd709a723beae9a4e681696874932db7aee2c (sha256:5040bd2983909aa8896b9932438c3f1479d25ae837a5f6220242a264d0221f2d) in docker-registry.cobrowser.io/nginx-edge"
dockerd[37603]: level=debug msg="Pushing layer: sha256:1941ca4a7a8446a0dd9d1572898c1d09e590b2de313880e41cc00cf02cd191ea"
dockerd[37603]: level=debug msg="Pushing layer: sha256:a2ae92ffcd29f7ededa0320f4a4fd709a723beae9a4e681696874932db7aee2c"
dockerd[37603]: level=debug msg="Assembling tar data for 3c85a3784a7b2e0dc74df35e05823e2bfd2f4b311f03d00d1b5299201727ae94"
dockerd[37603]: level=debug msg="Assembling tar data for f11bbbd4d01068bee1cc89214bae2ac5bba8c1e84600e710941289591634d108"
dockerd[37603]: level=error msg="Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=error msg="Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=debug msg="Pushing layer: sha256:1941ca4a7a8446a0dd9d1572898c1d09e590b2de313880e41cc00cf02cd191ea"
dockerd[37603]: level=debug msg="Assembling tar data for 3c85a3784a7b2e0dc74df35e05823e2bfd2f4b311f03d00d1b5299201727ae94"
dockerd[37603]: level=error msg="Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=debug msg="Pushing layer: sha256:a2ae92ffcd29f7ededa0320f4a4fd709a723beae9a4e681696874932db7aee2c"
dockerd[37603]: level=debug msg="Assembling tar data for f11bbbd4d01068bee1cc89214bae2ac5bba8c1e84600e710941289591634d108"
dockerd[37603]: level=error msg="Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=debug msg="Pushing layer: sha256:1941ca4a7a8446a0dd9d1572898c1d09e590b2de313880e41cc00cf02cd191ea"
dockerd[37603]: level=debug msg="Assembling tar data for 3c85a3784a7b2e0dc74df35e05823e2bfd2f4b311f03d00d1b5299201727ae94"
dockerd[37603]: level=error msg="Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=debug msg="Pushing layer: sha256:a2ae92ffcd29f7ededa0320f4a4fd709a723beae9a4e681696874932db7aee2c"
dockerd[37603]: level=debug msg="Assembling tar data for f11bbbd4d01068bee1cc89214bae2ac5bba8c1e84600e710941289591634d108"
dockerd[37603]: level=error msg="Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=debug msg="Pushing layer: sha256:1941ca4a7a8446a0dd9d1572898c1d09e590b2de313880e41cc00cf02cd191ea"
dockerd[37603]: level=debug msg="Assembling tar data for 3c85a3784a7b2e0dc74df35e05823e2bfd2f4b311f03d00d1b5299201727ae94"
dockerd[37603]: level=error msg="Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=debug msg="Pushing layer: sha256:a2ae92ffcd29f7ededa0320f4a4fd709a723beae9a4e681696874932db7aee2c"
dockerd[37603]: level=debug msg="Assembling tar data for f11bbbd4d01068bee1cc89214bae2ac5bba8c1e84600e710941289591634d108"
dockerd[37603]: level=error msg="Upload failed, retrying: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=debug msg="Pushing layer: sha256:1941ca4a7a8446a0dd9d1572898c1d09e590b2de313880e41cc00cf02cd191ea"
dockerd[37603]: level=debug msg="Assembling tar data for 3c85a3784a7b2e0dc74df35e05823e2bfd2f4b311f03d00d1b5299201727ae94"
dockerd[37603]: level=error msg="Upload failed: file integrity checksum failed for \"var/log/alternatives.log\""
dockerd[37603]: level=info msg="Attempting next endpoint for push after error: file integrity checksum failed for \"var/log/alternatives.log\""

This is our docker info:

Server Version: 17.05.0-ce
 Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 309
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.812GiB
Name: buildserver
ID: BASJ:CQJS:HD67:2GGF:ANGW:DIFW:3APG:CZ4J:LK3F:IPOD:3FZC:SO2N
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Answer 1

This solved a similar problem for us:

docker system prune -a