Artifact caching for multistage docker builds

Question

I have a Dockerfiles like this

# build-home
FROM node:10 AS build-home
WORKDIR /usr/src/app
COPY /home/package.json /home/yarn.lock /usr/src/app/
RUN yarn install
COPY ./home ./
RUN yarn build

# build-dashboard
FROM node:10 AS build-dashboard
WORKDIR /usr/src/app
COPY /dashboard/package.json /dashboard/yarn.lock /usr/src/app/
RUN yarn install
COPY ./dashboard ./
RUN yarn build

# run
FROM nginx
EXPOSE 80
COPY nginx.conf /etc/nginx/nginx.conf
COPY --from=build-home /usr/src/app/dist /usr/share/nginx/html/home
COPY --from=build-dashboard /usr/src/app/dist /usr/share/nginx/html/dashboard

Here building two react application and then artifacts of build are put in nginx. To improve build performance, I need to cache the dist folder in the build-home andbuild-dashboard build-stages. For this i create a volume in docker-compose.yml

...
  web:
    container_name: web
    build:
      context: ./web
    volumes:
      - ./web-build-cache:/usr/src/app
    ports:
      - 80:80
    depends_on:
      - api

I’ve stopped at this stage because I don’t understand how to add volume created bydocker-compose first for the build-home stage, and after adding thisvolume to the build-dashboard. Maybe i should be create a two volumes and attach each to each of build stages, but how do this?

UPDATE:

Initial build.

Home application:

1. Install modules: 100.91s
2. Build app: 39.51s

Dashboard application:

1. Install modules: 100.91s
2. Build app: 50.38s

Overall time:

real    8m14.322s
user    0m0.560s
sys     0m0.373s

Second build (without code or dependencies change):

Home application:

1. Install modules: Using cache
2. Build app: Using cache

Dashboard application:

1. Install modules: Using cache
2. Build app: Using cache

Overall time:

real    0m2.933s
user    0m0.309s
sys     0m0.427s

Third build (with small change in code in first app):

Home application:

1. Install modules: Using cache
2. Build app: 50.04s

Dashboard application:

1. Install modules: Using cache
2. Build app: Using cache

Overall time:

real    0m58.216s
user    0m0.340s
sys     0m0.445s

Initial build of home application without Docker: 89.69s

real    1m30.111s
user    2m6.148s
sys     2m17.094s

Second build of home application without Docker, the dist folder exists on disk (without code or dependencies change): 18.16s

real    0m18.594s
user    0m20.940s
sys     0m2.155s

Third build of home application without Docker, the dist folder exists on disk (with small change in code): 20.44s

real    0m20.886s
user    0m22.472s
sys     0m2.607s

In the docker-container, the third builds of the application is 2 times longer. This shows that if the result of the first build is on disk, other builds completed faster. In the docker container, all assemblies after the first are executed as long as the first, because there is no dist folder.

Answer 1

If you're using multi-stage builds then there's a problem with docker cache. The final image don't have layers with build steps. By using --target and --cache-from together you can save this layers and reuse them in rebuild.

You need something like

docker build \
  --target build-home \
  --cache-from build-home:latest \
  -t build-home:latest 


docker build \
  --target build-dashboard \
  --cache-from build-dashboard:latest \
  -t build-dashboard:latest 


docker build \
  --cache-from build-dashboard:latest \
  --cache-from build-home:latest \
  -t my-image:latest \

You can find more details at https://andrewlock.net/caching-docker-layers-on-serverless-build-hosts-with-multi-stage-builds---target,-and---cache-from/

Answer 2

You can't use volumes during image building, and in any case Docker already does the caching you're asking for. If you leave your Dockerfile as-is and don't try to add your code in volumes in the docker-compose.yml, you should get caching of the built Javascript files access rebuilds as you expect.

When you run docker build, Docker looks at each step in turn. If the input to the step hasn't changed, the step itself hasn't changed, and any files that are being added haven't changed, then Docker will just reuse the result of running that step previously. In your Dockerfile, if you only change the nginx config, it will skip over all of the Javascript build steps and reuse their result from the previous time around.

(The other relevant technique, which you already have, is to build applications in two steps: first copy in files like package.json and yarn.lock that name dependencies, and install dependencies; then copy in and build your application. Since the "install dependencies" step is frequently time-consuming and the dependencies change relatively infrequently, you want to encourage Docker to reuse the last build's node_modules directory.)