Disable cache for specific RUN commands

Question

I have a few RUN commands in my Dockerfile that I would like to run with -no-cache each time I build a Docker image.

I understand the docker build --no-cache will disable caching for the entire Dockerfile.

Is it possible to disable cache for a specific RUN command?

This can be useful during debugging.

For example, if a Dockerfile fails on a particular line, it may be useful to run all previous lines up to a certain point, while maintaining the cache of some of those previous lines.

The reason being some Docker commands can take a long time (perhaps several minutes?) to run.

Answer 1

Use --no-cache-filter to disable docker cache while running docker build for each target. For example,

docker build --no-cache-filter install --no-cache-filter rebuild .

one of my dockerfile has the following lines:

RUN yarn install --no-cache --network-timeout 1000000 && echo "installed package"
RUN npm rebuild node-sass && echo "rebuild node-sass"

also look into their offical docs for more ways to invalidate docker cache.

Answer 2

Divide your dockerfile in stage and put the RUN command under a specific stage. When you run the docker build command, add "--no-cache-filter " change with

Answer 3

I really hope this helps someone who reads this far down. I was having the opposite problem - RUN directives were not being cached when I really needed them to. The problem was that I had inadvertently implemented the solution explained above, but couldn't see it.

For our monorepo, we were using a build arg (SCOPE) to specify which project we wanted to build. We like to keep all of our ARG directives near the top. However, was invalidating everything below it... and it was very difficult to see. Let me show by code:

FROM ...
ARG SCOPE
# the next line was not getting cached
RUN echo "hello" 
RUN echo "${SCOPE}"

But making this subtle change worked:

FROM ...
# now it gets cached
RUN echo "hello"
ARG SCOPE
RUN echo "${SCOPE}"

Answer 4

Building on @Vladislav’s solution above I used in my Dockerfile

ARG CACHEBUST=0

to invalidate the build cache from hereon.

However, instead of passing a date or some other random value, I call

docker build --build-arg CACHEBUST=`git rev-parse ${GITHUB_REF}` ...

where GITHUB_REF is a branch name (e.g. main) whose latest commit hash is used. That means that docker’s build cache is being invalidated only if the branch from which I build the image has had commits since the last run of docker build.

Answer 5

If your goal is to include the latest code from Github (or similar), one can use the Github API (or equivalent) to fetch information about the latest commit using an ADD command.
docker build will always fetch an URL from an ADD command, and if the response is different from the one received last time docker build ran, it will not use the subsequent cached layers.

eg.

ADD "https://api.github.com/repos/username/repo_name/commits?per_page=1" latest_commit
RUN curl -sLO "https://github.com/username/repo_name/archive/main.zip" && unzip main.zip

Answer 6

I believe that this is a slight improvement on @steve's answer, above:

RUN git clone https://sdk.ghwl;erjnv;wekrv;qlk@gitlab.com/your_name/your_repository.git

WORKDIR your_repository

# Calls for a random number to break the cahing of the git clone
# (https://stackoverflow.com/questions/35134713/disable-cache-for-specific-run-commands/58801213#58801213)
ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache
RUN git pull

This uses the Docker cache of the git clone, but then runs an uncached update of the repository.

It appears to work, and it is faster - but many thanks to @steve for providing the underlying principles.

Answer 7

There's always an option to insert some meaningless and cheap-to-run command before the region you want to disable cache for.

As proposed in this issue comment, one can add a build argument block (name can be arbitrary):

ARG CACHEBUST=1 

before such region, and modify its value each run by adding --build-arg CACHEBUST=$(date +%s) as a docker build argument (value can also be arbitrary, here it is current datetime, to ensure its uniqueness across runs).

This will, of course, disable cache for all following blocks too, as hash sum of the intermediate image will be different, which makes truly selective cache disabling a non-trivial problem, taking into account how docker currently works.

Answer 8

Use

ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache

before the RUN line you want to always run. This works because ADD will always fetch the file/URL and the above URL generates random data on each request, Docker then compares the result to see if it can use the cache.

I have also tested this and works nicely since it does not require any additional Docker command line arguments and also works from a Docker-compose.yaml file :)

Answer 9

the feature added a week ago.

ARG FOO=bar

FROM something
RUN echo "this won't be affected if the value of FOO changes"
ARG FOO
RUN echo "this step will be executed again if the value of FOO changes"

FROM something-else
RUN echo "this won't be affected because this stage doesn't use the FOO build-arg"

https://github.com/moby/moby/issues/1996#issuecomment-550020843

Answer 10

Another quick hack is to write some random bytes before your command

RUN head -c 5 /dev/random > random_bytes && <run your command>

writes out 5 random bytes which will force a cache miss

Answer 11

As of February 2016 it is not possible.

The feature has been requested at GitHub

Answer 12

Not directly but you can divide your Dockerfile in several parts, build an image, then FROM thisimage at the beginning of the next Dockerfile, and build the image with or without caching