How to cache the RUN npm install instruction when docker build a Dockerfile

Question

I am currently developing a Node backend for my application. When dockerizing it (docker build .) the longest phase is the RUN npm install. The RUN npm install instruction runs on every small server code change, which impedes productivity through increased build time.

I found that running npm install where the application code lives and adding the node_modules to the container with the ADD instruction solves this issue, but it is far from best practice. It kind of breaks the whole idea of dockerizing it and it cause the container to weight much more.

Any other solutions?

Answer 1

Docker now provides a native way (cache mounts) to handle this:

   FROM node:latest
   WORKDIR /app
   RUN --mount=type=cache,target=/root/.npm npm install

In this example, the npm install command uses a cache mount for the /root/.npm directory, the default location for the npm cache. The cache mount is persisted across builds, so even if you end up rebuilding the layer, you only download new or changed packages. Any changes to the cache are persisted across builds, and the cache is shared between multiple builds. https://docs.docker.com/build/cache/optimize/#use-cache-mounts

Answer 2

I wanted to use volumes, not copy, and keep using docker compose, and I could do it chaining the commands at the end

FROM debian:latest
RUN apt -y update \
    && apt -y install curl \
    && curl -sL https://deb.nodesource.com/setup_12.x | bash - \
    && apt -y install nodejs
RUN apt -y update \
    &&  apt -y install wget \
        build-essential \
        net-tools
RUN npm install pm2 -g      

RUN mkdir -p /home/services_monitor/ && touch /home/services_monitor/
RUN chown -R root:root /home/services_monitor/

WORKDIR /home/services_monitor/

CMD npm install \
    && pm2-runtime /home/services_monitor/start.json

Answer 3

I've found that the simplest approach is to leverage Docker's copy semantics:

The COPY instruction copies new files or directories from and adds them to the filesystem of the container at the path .

This means that if you first explicitly copy the package.json file and then run the npm install step that it can be cached and then you can copy the rest of the source directory. If the package.json file has changed, then that will be new and it will re-run the npm install caching that for future builds.

A snippet from the end of a Dockerfile would look like:

# install node modules
WORKDIR  /usr/app
COPY     package.json /usr/app/package.json
RUN      npm install

# install application
COPY     . /usr/app

Answer 4

you don't need to use tmp folder, just copy package.json to your container's application folder, do some install work and copy all files later.

COPY app/package.json /opt/app/package.json
RUN cd /opt/app && npm install
COPY app /opt/app

Answer 5

Weird! No one mentions multi-stage build.

# ---- Base Node ----
FROM alpine:3.5 AS base
# install node
RUN apk add --no-cache nodejs-current tini
# set working directory
WORKDIR /root/chat
# Set tini as entrypoint
ENTRYPOINT ["/sbin/tini", "--"]
# copy project file
COPY package.json .

#
# ---- Dependencies ----
FROM base AS dependencies
# install node packages
RUN npm set progress=false && npm config set depth 0
RUN npm install --only=production 
# copy production node_modules aside
RUN cp -R node_modules prod_node_modules
# install ALL node_modules, including 'devDependencies'
RUN npm install

#
# ---- Test ----
# run linters, setup and tests
FROM dependencies AS test
COPY . .
RUN  npm run lint && npm run setup && npm run test

#
# ---- Release ----
FROM base AS release
# copy production node_modules
COPY --from=dependencies /root/chat/prod_node_modules ./node_modules
# copy app sources
COPY . .
# expose port and define CMD
EXPOSE 5000
CMD npm run start

Awesome tuto here: https://codefresh.io/docker-tutorial/node_docker_multistage/

Answer 6

Ok so I found this great article about efficiency when writing a docker file.

This is an example of a bad docker file adding the application code before running the RUN npm install instruction:

FROM ubuntu

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update
RUN apt-get -y install python-software-properties git build-essential
RUN add-apt-repository -y ppa:chris-lea/node.js
RUN apt-get update
RUN apt-get -y install nodejs

WORKDIR /opt/app

COPY . /opt/app
RUN npm install
EXPOSE 3001

CMD ["node", "server.js"]

By dividing the copy of the application into 2 COPY instructions (one for the package.json file and the other for the rest of the files) and running the npm install instruction before adding the actual code, any code change wont trigger the RUN npm install instruction, only changes of the package.json will trigger it. Better practice docker file:

FROM ubuntu
MAINTAINER David Weinstein <david@bitjudo.com>

# install our dependencies and nodejs
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update
RUN apt-get -y install python-software-properties git build-essential
RUN add-apt-repository -y ppa:chris-lea/node.js
RUN apt-get update
RUN apt-get -y install nodejs

# use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
COPY package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir -p /opt/app && cp -a /tmp/node_modules /opt/app/

# From here we load our application's code in, therefore the previous docker
# "layer" thats been cached will be used if possible
WORKDIR /opt/app
COPY . /opt/app

EXPOSE 3000

CMD ["node", "server.js"]

This is where the package.json file added, install its dependencies and copy them into the container WORKDIR, where the app lives:

ADD package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir -p /opt/app && cp -a /tmp/node_modules /opt/app/

To avoid the npm install phase on every docker build just copy those lines and change the ^/opt/app^ to the location your app lives inside the container.