Docker docker-compose not picking up relevant cached image

Question

I ran docker-compose build celery and (after hours of attempts on my poor connection) it succeeded. The first 80% of the app Dockerfile is identical, yet it doesn't reuse the cache. From what I can tell browsing around, Docker compares the base image and the instructions in Dockerfile and reuses if possible.

---

UPDATE: The problem this question refers to disappeared, no idea why. Notes below.

---

Yet I'm getting:

docker-compose build celery
Building celery
Step 1 : FROM python:2.7
 ---> eb867117097c
Step 2 : RUN apt-get update && apt-get install -y vim gdal-bin libgdal-dev postgresql-client
 ---> Using cache
 ---> 2966946ca235

. . . identical steps . . .

Step 9 : RUN pip install --no-cache-dir -r requirements/production.txt
 ---> Running in 02b42f721a34
Collecting git+https://github.com . . .
. . .
 ---> f70ecc01cada
Removing intermediate container 02b42f721a34
Step 10 : RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 ---> Running in 3575383edcef

When I type docker images I can see f70ecc01cada, but it is not found in the following:

docker-compose build app
Building app
Step 1 : FROM python:2.7
 ---> eb867117097c
Step 2 : RUN apt-get update && apt-get install -y vim gdal-bin libgdal-dev postgresql-client
 ---> Using cache
 ---> 03e5040df047

. . . identical steps . . .

Step 9 : RUN pip install --no-cache-dir -r requirements/production.txt
 ---> Running in bb26fab28548
Collecting git+https://github.com . . .
. . .

Am I right in thinking the latter should reuse the f70ecc01cada cached image? [Edit: I now realize 03e5040df047 is just a container thanks @Ohmen, so no point deleting it]

I have tried editing docker-compose to be identical memory etc but it didn't help. Can I force reuse?

---

UPDATE: The app build all of a sudden started picking up the cache from the celery build. I don't know what fixed it. Between it failing and working, I did connect to the image on the celery hierarchy just before the Dockerfiles diverged, manually ran the remaining commands on the app image (docker run -it ... --entry=bash, adduser and chown app/), and docker committed the resulting container. I then re-ran docker-compose build app and for some reason it picked up the cache from the celery hierarchy correctly and ran the remaining commands (create user and chown) successfully. I have no idea what made it work all of a sudden.

---

EDIT: Docker history of the two image trees, showing that the history diverged on the line RUN apt-get update && apt-get install -y vim gdal-bin libgdal-dev postgresql-client. Maybe I ran both builds simultaneously? I was pretty sure I didn't, as on my connection they take an hour or more, but perhaps that is the boring and embarrassing explanation. Perhaps Docker was randomly selecting the tree that subsequently failed, then randomly (after 56 attempts) selected the one that didn't.

Initial successful build - layers sequenced from bottom up:

Chriss-MacBook-Pro:~ technicaltitch$ docker history fdw_celery
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d71bf8f2b902        28 hours ago        /bin/sh -c #(nop)  ENTRYPOINT ["/bin/sh" "-c"   0 B                 
336bdb2b9ca9        28 hours ago        /bin/sh -c #(nop)  USER [celery]                0 B                 
a1259f89bc1f        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   15.39 MB            
57dd9a330337        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   335.2 kB            
3accad8aa55c        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   0 B                 
b4ff0c1d71fb        28 hours ago        /bin/sh -c #(nop) COPY dir:a8f922c5264fe2275a   15.39 MB            
bc65bc84abbc        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   0 B                 
f70ecc01cada        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   214.1 MB            
85eb413bd5da        28 hours ago        /bin/sh -c #(nop)  ARG PIP_INDEX_URL=https://   0 B                 
3cee97bf3ce1        28 hours ago        /bin/sh -c #(nop)  ARG PIP_TRUSTED_HOST=127.0   0 B                 
cb090c4b1886        28 hours ago        /bin/sh -c #(nop)  WORKDIR /usr/src/app         0 B                 
d0a1d3d15f94        28 hours ago        /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d   3.145 kB            
e27ecd707562        28 hours ago        |1 http_proxy= /bin/sh -c mkdir -p /usr/src/a   0 B                 
0b6e2857206c        28 hours ago        |1 http_proxy= /bin/sh -c VERSION=$(gdal-conf   5.903 MB            
2966946ca235        29 hours ago        |1 http_proxy= /bin/sh -c apt-get update && a   241.4 MB            
eb867117097c        42 hours ago        /bin/sh -c #(nop) CMD ["python2"]               0 B                 

Second, failing build, with same script:

Chriss-MacBook-Pro:~ technicaltitch$ docker history 5ee
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
5ee29875a771        29 hours ago        /bin/sh -c #(nop)  ARG PIP_INDEX_URL=https://   0 B                 
daad37fd701b        29 hours ago        /bin/sh -c #(nop)  ARG PIP_TRUSTED_HOST=127.0   0 B                 
f891ee9277da        29 hours ago        /bin/sh -c #(nop)  WORKDIR /usr/src/app         0 B                 
2cce4acf9f2f        29 hours ago        /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d   3.145 kB            
9484e6b7fa51        29 hours ago        |1 http_proxy=None /bin/sh -c mkdir -p /usr/s   0 B                 
0031cdd56926        29 hours ago        |1 http_proxy=None /bin/sh -c VERSION=$(gdal-   5.903 MB            
03e5040df047 <-- [already diverged]      |1 http_proxy=None /bin/sh -c apt-get update    243.1 MB            
eb867117097c        42 hours ago        /bin/sh -c #(nop) CMD ["python2"]               0 B                 

Finally, this is the image history after it magically succeeded. I had not deleted the other images when this worked (and still haven't):

Chriss-MacBook-Pro:~ technicaltitch$ docker history fdw_app
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
edc615da4d6f        26 hours ago        /bin/sh -c #(nop)  ENTRYPOINT ["/usr/src/app/   0 B                 
747c8744f6a7        26 hours ago        /bin/sh -c #(nop)  EXPOSE 8000/tcp              0 B                 
8c543333e10b        26 hours ago        /bin/sh -c #(nop)  USER [django]                0 B                 
b08f02d80d29        26 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   15.39 MB            
ba42415ad78b        26 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   335.2 kB            
027e0c8e39a9        26 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   0 B                 
d56a78c02d18        26 hours ago        /bin/sh -c #(nop) COPY dir:9274abe4540edd1e86   15.39 MB            
bc65bc84abbc        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   0 B                 
f70ecc01cada        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   214.1 MB            
85eb413bd5da        28 hours ago        /bin/sh -c #(nop)  ARG PIP_INDEX_URL=https://   0 B                 
3cee97bf3ce1        28 hours ago        /bin/sh -c #(nop)  ARG PIP_TRUSTED_HOST=127.0   0 B                 
cb090c4b1886        28 hours ago        /bin/sh -c #(nop)  WORKDIR /usr/src/app         0 B                 
d0a1d3d15f94        28 hours ago        /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d   3.145 kB            
e27ecd707562        28 hours ago        |1 http_proxy= /bin/sh -c mkdir -p /usr/src/a   0 B                 
0b6e2857206c        28 hours ago        |1 http_proxy= /bin/sh -c VERSION=$(gdal-conf   5.903 MB            
2966946ca235 <-- [correct img, suddenly..why?!]  ..xy= /bin/sh -c apt-get update && a   241.4 MB            
eb867117097c        42 hours ago        /bin/sh -c #(nop) CMD ["python2"]               0 B                 

The following is the start of both Dockerfiles. I have tried making the machine definitions in docker-compose.yml identical but to no avail (possibly after the 03e image was created).

FROM python:2.7

RUN apt-get update && apt-get install -y vim gdal-bin libgdal-dev postgresql-client

# On Jessie, $(gdal-config --version) gives 1.10.1 and Pypi only has 1.10.0
# so we need to strip the last part of the version number
RUN VERSION=$(gdal-config --version); CFLAGS=$(gdal-config --cflags) easy_install GDAL==${VERSION%.*}

RUN mkdir -p /usr/src/app/requirements
COPY requirements /usr/src/app/requirements

WORKDIR /usr/src/app

ARG PIP_TRUSTED_HOST=127.0.0.1
ARG PIP_INDEX_URL=https://pypi.python.org/simple/
RUN pip install --no-cache-dir -r requirements/production.txt

# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY . /usr/src/app

RUN mkdir -p /usr/src/app/log

Answer 1

COPY requirements /usr/src/app/requirements

If you change ANY file in this directory it will invalidate the cache from this point onward. This includes tmp files created by your editor, unless you put them in your .dockerignore file.

ARG PIP_TRUSTED_HOST=127.0.0.1 ARG PIP_INDEX_URL=https://pypi.python.org/simple/

If you change either of these ARG it will invalidate the cache from this point onward.

My guess is that one of these changes, causing it to skip the cache.

Answer 2

The ---> 03e5040df047 is just a temporary name for the container this does not indicate diferent images...

Docker uses the cache if the base image is known and the commands are the same. For ADD od COPY also the checksum of the files has to match as well. As if I can tell docker uses the cache in your snippet.

You can only force docker to not use the cache.