CODEWITHSUNDEEP
npmnugetazure-devops
matcheek
matcheek

Reputation: 5147

VSTS multi-phased builds - run nuget restore and npm install in parallel

I am having a build where in pre-compilation stage nuget restore is taking ~3 minutes to restore packages from cache and so does npm.

These two restoration from caches could run in parallel but I am not clear whether this is possible using the VSTS Phases.

Each phase may use different agents. You should not assume that the state from an earlier phase is available during subsequent phases.

What I would need is a way to pass the content of packages and node_modules directories from two different phases into a third one that invokes the compiler.

Is this possible with VSTS phases?

Upvotes: 1

Views: 356

Answers (2)

matcheek
matcheek

Reputation: 5147

A real problem here wasn't that VSTS hosted agent npm install and nuget restore could not have been run in parallel on a hosted agent. No. A real problem was that hosted agent do not use nuget cache by design.

We have determined that this issue is not a bug. Hosted agent will download nuget packages every time you queue a new build. You could not speed this nuget restore step using a hosted agent.

https://developercommunity.visualstudio.com/content/problem/148357/nuget-restore-is-slow-on-hostedagent-2017.html

So a solution to take nuget restore time from 240s to 20s was to move it to a local agent. That way local cache do get used.

Upvotes: 0

Daniel Mann
Daniel Mann

Reputation: 58980

I wouldn't do this with phases. I'd consider not doing it at all. Restoring packages (regardless of type) is an I/O bound operation -- you're not likely to get much out of parallelizing it. In fact, it may be slower. The bulk of the time spent restoring packages is either waiting for a file to download, or copying files around on disk. Downloading twice as many files just takes twice as long. Copying two files at once takes double the time. That's roughly speaking, of course -- it may be a bit faster in some cases, but it's not likely to be significantly faster for the average case.

That said, you could write a script to spin off two separate jobs and wait for them to complete. Something like this, in PowerShell:

$dotnetRestoreJob = (Start-Job -ScriptBlock { dotnet restore } ).Id
$npmRestoreJob = (Start-Job -ScriptBlock { npm install } ).Id

do {
    $jobStatus = Get-Job -Id @($dotnetRestoreJob, $npmRestoreJob)
    $jobStatus
    Start-Sleep -Seconds 1
}
while ($jobStatus | where { $_.State -eq 'Running' })

Of course, you'd probably want to capture the output from the jobs and check for whether there was a success exit code or a failure exit code, but that's the general idea.

Upvotes: 2

Related Questions