CODEWITHSUNDEEP
node.jscachingnpmcontinuous-integration
Nedudi
Nedudi

Reputation: 5949

Is there a way to speedup npm ci using cache?

for now npm ci is the most common way to install node modules when using CI. But it is honestly really slow. Is there a way to speedup npm ci using cache or do not fully remove existing packages (whole node_modules folder)?

Upvotes: 88

Views: 86087

Answers (7)

AD95
AD95

Reputation: 180

Although this doesn't exactly answer the question, but since you asked "a way to speed up npm build without fully removing existing npm modules". I would like to share a solution I took to reduce build time in production. This was done using cloud caching techniques. Points I considered were as follows

1. If no change in package.json, don't install npm_modules & fetch it from cache
2. Set up a expiry limit for cache, so even if there is no change in package.json for the time limit you have set, revalidate cache
   and install modules again.

Cloud : GCP

Leveraged GCP kaniko cache with expiry limit configured for npm modules & added this as part of docker build process in deployment. This saved probably 2-3 mins from my deployment cycle.

Upvotes: 0

Thành Trung Phạm
Thành Trung Phạm

Reputation: 139

You can set your cache folder when running with npm ci

npm ci --cache .npm

and then in your ci, cache the .npm folder

Upvotes: 6

Artem
Artem

Reputation: 1870

Disclaimer, this method is not reliable enough to use for final builds.

However, if you need to run some auxiliary script as quickly as possible which just requires all packages from packages-lock.json, you can simply compare it with the .package-lock.json file in the node_modules folder. And if changes are detected there, then call the usual npm ci. In my case, this has made some tasks speed up hundreds of times.

Here's a link to the gist - install.js. This works in npm v7.0 and newer.

Upvotes: 0

Sam Lahm
Sam Lahm

Reputation: 93

This was helpful for me, though it's probably only useful locally https://stackoverflow.com/a/61364681/9727824

Basically using npm ci --production which will skip installing dev-dependencies.

Edit: later versions of NPM suggest to use npm ci --omit=dev as described here.

Upvotes: 3

pixelbrackets
pixelbrackets

Reputation: 2318

tl;dr ̶N̶o̶.̶ A little.

npm ci should be preferred in CI because it respects the package-lock.json file. Unlike npm install, which rewrites the file and always installs new versions.

By design this command always purges all local packages, by removing the node_modules directory in the beginning. This is the main reason for long builds. And there is no option to avoid this irritating behaviour.

On a local machine you may speed up npm ci by adding the option --prefer-offline, which tells NPM to ignore the cache minimum time and use locally cached packages right away instead of verifying them against the registry.

However, the NPM cache is located in ~/.npm on Unix, or %AppData%/npm-cache on Windows. These folders are not cacheable by default in most CIs. For example GitLab CI caches only have the repository as available workspace. Other directories like the home directory or system directories (eg. apt) are not cached. Therefore this setting probably wont affect your CI build time.

In older version of NPM the option --progress=false had a significant effect on build time, by removing progress bars. This issue seems to be gone however, I can not measure a noteworthy difference anymore.

A best practice and definitely a gain in speed is to separate packages into production and development. By passing the option --only=production NPM will then ignore development dependencies. Due to the reasons above this wont affect caching.

Update 2021-06: It is now possible to change the cache directory

As pointed out by the comments below, it is now possible to change the location of NPM's cache directory. For example by passing it as argument on every command (--cache .npm) or environment variable (npm_config_cache=.npm). So change this to a path to a temporary directory in your repository, add it to the CI cache stack but exclude it from deployment builds. Then you will be able to use the --prefer-offline argument in CI scripts as well.

Summary:

  • Split dependencies
  • Set up a local cache directory
  • Use npm ci --cache <local cache directory> --prefer-offline --only=production --silent
  • This will speed up the process, but due to the command design it can't be as fast as npm install

Upvotes: 57

Philippe GRANET
Philippe GRANET

Reputation: 579

You could tell your CI to cache npm's cache dir, and then use the options --prefer-offline and --no-audit, example:
npm ci --prefer-offline --no-audit

Upvotes: 12

Kiliandeca
Kiliandeca

Reputation: 716

NPM cache is located in ~/.npm but in most CIs you can only cache things inside your working directory.

What you can do to circumvent this is changing the cache directory to your current directory with npm set cache .npm. The NPM cache will now be located in ./.npm an you can cache this folder between CI jobs.

Example with GitLab CI:

my-super-job:
  image: node:13-alpine
  script:
    - npm set cache .npm
    - npm ci 
  cache:
    paths:
      - .npm

EDIT: Just discovered that you can set the config as a command line flag so npm ci --cache .npm should do the same

Upvotes: 70

Related Questions