CODEWITHSUNDEEP
javascriptnode.jsdockernpm
paul23
paul23

Reputation: 9455

Npm install being slow and lots of "cache miss" after nearly every library

While everything seems to work on my local computer;

Whenever I run nodejs inside a docker (docker run node:18) and I clone a project, type npm install to get all libraries and work with them, it is really slow. Like 10 seconds slow.

While it works quickly after this initial bump I also notice that each library (as far as I can tell) has a (cache miss) appended after the timing (which is around 10 seconds). What is happening is this a problem/can I fix it?

Just to stress: it happens in any docker, whether I use node-alpine, node docker or even just an ubuntu docker and install node there manually.

After some hints from @NaorTedgi I notice that this is indeed due to the fact that the package is situated outside the docker and linked through a volume. I also notice that the timing itself (the 15 seconds) is dependent on the amount of packages it tries to load. With a single package it's too fast too notice and with a few it's only half a second.

So to test it the following steps I take:

make a new directory (~/javascript-test) and put the following into a package.json file:

{
  "name": "backend",
  "version": "1.0.0",
  "description": "",
  "main": "app.js",
  "scripts": {
    "start": "NODE_ENV=production node ./javascript/app.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "bcryptjs": "^2.4.3",
    "connect-redis": "^6.1.3",
    "cookie-parser": "^1.4.6",
    "date-fns": "^2.28.0",
    "debug": "^4.3.3",
    "express": "^4.17.2",
    "express-session": "^1.17.2",
    "http-errors": "^2.0.0",
    "knex": "^2.1.0",
    "morgan": "^1.10.0",
    "multer": "^1.4.5-lts.1",
    "nanoid": "^3.3.4",
    "node-cron": "^3.0.1",
    "objection": "^3.0.1",
    "pg": "^8.7.1",
    "redis": "^4.2.0",
    "typescript": "^4.7.4",
    "uuid": "^8.3.2"
  }
}

Open shell to this directory and run (to initialize package.json)

npm install
rm -rf node_modules

Then run the docker with the volume (obviously with sudo if required):

docker run --name node-test --rm -it -v ~/javascript-test:/javascript node:18

Open a second shell (since the default entrypoint isn't sh from the node dockers) and execute:

docker exec -it node-test sh

Inside the docker shell:

cd javascript && npm install

With these steps I notice cache misses after 2 seconds.

Finally I notice that if I do remove node_modules and reinstall the modules (npm install) a second time inside the docker no cache misses happen. So to test a second time one has to end the node docker and rerun it (docker run... in first shell).

For those who like a dockerfile, this is the simplest file that still exhibit the error (once again make sure to bind the volume containing above `package.json` and the corresponding `package-lock.json`) 
FROM node:18
WORKDIR /javascript
ENTRYPOINT npm install

A git repository of the Dockerfile and the package.json/package-lock.json files: https://github.com/pulli23/docker-npm-test

run it through (if cloned into ~/dockertest)

sudo docker build -t nodetest .  && sudo docker run --name node-test --rm -it -v ~/dockertest/javascript-test:/javascript nodetest

Upvotes: 12

Views: 20629

Answers (3)

Naveen
Naveen

Reputation: 1

I also faced cache miss while running npm install, it got resolved by not doing a npm cache clear.

Just i cleared node modules and ran the npm install again.

i face the network Error , after cache miss. so i followed these steps

  1. rm -rf node modules
  2. npm install/npm install --force

After this it npm installed correctly.

Upvotes: 0

Naor Tedgi
Naor Tedgi

Reputation: 5717

few things you need to check:

  1. if you're using more then one registry make sure npm is your default

npm config set registry=https://registry.npmjs.com/

  1. if your project repository has package-lock.json
  • then make sure you're not using volume to the current directory node_modules dir to your image if so it will be much faster to delete node_modules before installation
  • run install with ci => npm ci
  1. make sure your package-lock.json is generated from the ‘npm i’ command from inside the container! Running npm i locally will generate libraries compatible with your local node version and in case of c++ add-ons generate dll,so or dylib file according to your OS

Upvotes: 8

Slava Kuravsky
Slava Kuravsky

Reputation: 2824

My guess it is an issue related to networking (f.e. proxy) or authentication (f.e. connection to enterprise repository). Your local connection has something what your container is missing, maybe some certificate.

You should go into the container and test if you can reach the repository.

Upvotes: 2

Related Questions