Problem We have a large number of build configurations in TeamCity that deal with npm installs of many of the same packages Because we need to enforce clean checkouts for a lot of our builds (e.g. all PRs), the node_modules folder within a given build will never be re-used The npm install portion of our build takes 4 minutes, which is around 40% of the time for a given build. Across hundreds of builds a day on 10 agents, this is a number worth reducing. Question What is the recommended way to centralize / re-use the node_modules folder between builds given our stack? Technology Stack Windows Server 2008 R2 Node LTS 4.x TeamCity 9.x What We've Tried Use npm cache -- This happens by default, so at least it's not like we're downloading packages from the internet every time. Symlinking node_modules to a common directory -- This approach worked (went from minutes to seconds) but unfortunately some gulp/grunt tasks fail when dealing with symlinks on Windows. Womp womp. Other Approaches We're Trying Create a node_modules folder further up the hierarchy : Not sure, but it seems like node may hierarchically search for the node_modules folder. In that case, creating a node_modules folder in a parent directory may solve the issue. Use the NODE_PATH environment variable -- Not sure if setting this will do something different similar to the point above and provide a common node_modules folder when one is not found in one of our builds. Use Powershell to Move-Item back & forth between a backup directory : If we have to hack it, it might be worth it to use a backup directory per project ID and use Move-Item , which should update partition links rather than copying files, at least. That approach is outlined on this blog post .

node.jsnpmteamcity

SeanKilleen

Reputation: 8977

How do I reduce the time for npm install across multiple TeamCity builds?

Problem

We have a large number of build configurations in TeamCity that deal with npm installs of many of the same packages
Because we need to enforce clean checkouts for a lot of our builds (e.g. all PRs), the node_modules folder within a given build will never be re-used
The npm install portion of our build takes 4 minutes, which is around 40% of the time for a given build.
Across hundreds of builds a day on 10 agents, this is a number worth reducing.

Question

What is the recommended way to centralize / re-use the node_modules folder between builds given our stack?

Technology Stack

Windows Server 2008 R2
Node LTS 4.x
TeamCity 9.x

What We've Tried

Use npm cache -- This happens by default, so at least it's not like we're downloading packages from the internet every time.
Symlinking node_modules to a common directory -- This approach worked (went from minutes to seconds) but unfortunately some gulp/grunt tasks fail when dealing with symlinks on Windows. Womp womp.

Other Approaches We're Trying

Create a node_modules folder further up the hierarchy: Not sure, but it seems like node may hierarchically search for the node_modules folder. In that case, creating a node_modules folder in a parent directory may solve the issue.
Use the NODE_PATH environment variable -- Not sure if setting this will do something different similar to the point above and provide a common node_modules folder when one is not found in one of our builds.
Use Powershell to Move-Item back & forth between a backup directory: If we have to hack it, it might be worth it to use a backup directory per project ID and use Move-Item, which should update partition links rather than copying files, at least. That approach is outlined on this blog post.

Upvotes: 6

Answers (3)

banyudu

Reputation: 1102

Have you tried docker cache?

Put your npm install in Dockerfile, run docker build

FROM node:12

WORKDIR /code

COPY ./package.json ./package-lock.json /code

RUN npm install # skiped when package.json/package-lock.json not changed.

ADD . /code

# do other things

Docker build creates a layer for each command(COPY/ADD/RUN), and use the layer as cache(if nothing changed) when build agagin.

As long as package.json/package-lock.json not changed

docker build will use cache for

COPY ./package.json ./package-lock.json /code

and

RUN npm install

so you don't need to speed up npm install, it just skiped.

Upvotes: 0