Nodejs npm step downloads packages on every build in TeamCity

Question

I'm a bit of a n00b when it comes to nodejs npm, but since implementing it in our build environment using steps recommended on several articles its tripled our build times.

We use it for the standard stuff (minify/concat/etc js/css/etc)

We use TeamCity and have added a Node.js NPM step then a gulp step to run the tasks (RE: https://github.com/jonnyzzz/TeamCity.Node)

The task to setup NPM takes the most time, 2min 10 seconds, which is over 65% of the total build time calling the command "npm install", which appears to re-download all the packages on each build

Step 3/7: NPM Setup (Node.js NPM) (2m:10s)

[npm install] Starting: cmd /c npm install

Out total build times before were around 1min 30sec, including unit tests.

is there anyway to cache these locally and prevent re-download on each build? in the user profile or something maybe as opposed to the build folder?

More detail..

This probably best explains the setup http://www.dotnetcurry.com/visualstudio/1096/using-grunt-gulp-bower-visual-studio-2013-2015

We have C# projects that are using the new Task Runner Explorer, Dependencies are saved into a package.json by this, you pre-run "npm install" once on your local environment in you workspace (need to use a .tfignore to prevent it from checking in to source) then not again, unless you start a new local workspace.

When the build run it needs to run "npm install" from the command line and it picks up dependencies from the package.json file and installs them into a sub folder inside the working directory of the build every time, even if the files are already there from a previous build(i.e. TC agent hasn't cleaned them up), afaik you cant install them outside the working folder.

I could be wrong... Or I should say I hope I'm wrong, and looking for a way for gulp to support this, but what ever way we make it work will need to work with task runner explorer so the F5 experience for the dev is still the same on their local.

We do have multiple agents yes.

Answer 1

For people who also struggling with this. Other reason might be cleaning out directory while switching branch.

Project Settings -> Version Control Settings -> VCS Root Settings (edit)

TeamCity default setting is cleaning up all untracked files on branch change (even ignored ones) that cause node_modules to be removed on every branch change.

Changing this settings to "All non-ignored untracked files" should help.

Answer 2

If you aren't using "loose" package versions this could be your issue. You can specify something like npm install angular-cli@0.1.0.0-beta.10, but then you are stuck on an older version all the time, and if your developers are frequently changing what they build against you will have churn on your build system.

In your packages.json you can use * and ~ to define wildcard versions, like 3.* or ~2.5 which would let you get any "minor" revision of 3.x like 3.0.2 or 3.0.99, and the 2.5 would be something similar, any version of 2.5.x could be installed, but not 2.6.0.

Answer 3

The best way i found to fix this was to backup/restore the node modules folder, i've done a blog post about it here

https://beerandserversdontmix.com/2016/06/04/teamcity-and-avoiding-redownloading-of-npm-packages/

Answer 4

I don't know about Node.js, but here are a couple TeamCity-specific suggestions:

1. Does NPM perhaps download the files into %TEMP%? If so, they won't be reusable between subsequent TeamCity builds because a TeamCity agent hijacks the %TEMP% directory (redirects it to <TeamCity Home>/buildAgent/temp/buildTmp) and always completely wipes this directory before every new build. (See buildTmp here.)
   • In that sense, it would be preferable if you could instruct NPM to store the downloaded files in the workspace (the directory where you checkout your build) instead.
2. If NPM is downloading into the workspace (the checkout dir), have you perhaps requested to do a clean checkout on every run? (See Edit Configuration Settings | Version Control Settings | Show advanced options | Clean all files in the checkout directory before the build checkbox.)
   • In that case, uncheck the checkbox.
3. Is perhaps TeamCity cleaning up the checkout directory due to low disk space? This clean-up kicks in automatically when TeamCity notices it's running out of space. (The clean-up can be made even more aggressive with the Free disk space build feature.)
   • In that case, stop using the build feature. If it's not used and the automatic clean-up is to blame, it's hard to control. It's best if you simply clean-up that part of your file-system which is not managed by TeamCity (your own %TEMP% and other places) and thus give some leeway to TeamCity.
4. Is your build running on a different agent every time? (Consult the build history.) If so, it cannot reuse the downloaded artifacts (even if they are downloaded into the checkout dir), since they are downloaded to a different machine's filesystem every time. I doubt this is the case though, since TeamCity gravitates towards agent-workspace reuse (sticking to the same agent).
   • In that case, you can force agent reuse by setting an agent requirement, specifying that you want your builds to run on one specific agent all the time. You can also single that agent out into its own pool, so that no other builds can run on it.