Is it advantageous to have multiple build agents on the same machine?

Question

We are using Team Foundation Build and currently have a single build agent.

If we configure a second build agent on the same machine, will this give us a performance benefit when we have a queue of builds to complete?

Note we are not using agent tagging, so all of our builds could be sent to either of the two agents.

If building is largely a CPU-intensive task, I am struggling to see that there would be a significant advantage in parallel builds, as the total processor time to build the queue would remain the same, regardless of which agent builds it.

However if there is also significant disk activity during a build, perhaps one agent could be making use of available CPU cycles while the other agent is accessing the disk?

There is an MSDN article that mentions different build topologies, and this does indeed have multiple build agents on the same machine in some cases. However it is not clear if this is only because the agents have different tags.

Answer 1

I was under the same impression... that it depends, but in the last couple of days I started encountering concurrency issues between the pipeline requirements, and I've discovered this is actually a very bad idea. One such example is using a sonar-scanner integrated with MsBuild... or any other such tool that adds a lock of a resource.

Answer 2

As Daniel already mentions, it depends, but generally, if you agent is doing plain builds, then having multiple agents will allow you to run multiple builds in parallel. When doing so IO and Memory constraints may cause issues on your build server, as may certain Anti-Virus products.

If you build server is equipped with fast local storage (ram disks, physical SSD's or physical drives for Virtual machines are preferred over SAN storage for the build workspaces) and has enough CPU cores assigned it may be able to run quite a few builds in parallel.

If configured, both MsBuild and Team Build can spread the load of building multiple solutions, projects and/or configurations over multiple agents, causing massive improvements in the time it takes to build your solution. (as long as the hardware can keep up with this).

Aside: Along the same lines, when your hardware can't keep up, having multiple agents on different servers allows you to scale out (vs scale up) your build to these multiple agents, reducing the overall build time that way.

Especially when you have a CI build configured and your build takes longer than a few minutes, it is beneficial to the development teams to have multiple agents, that way their builds run in parallel and all developers get their feedback faster. had you only had one build agent, it would cause your developers to have to wait for each other to finish.

There are indeed a few scenario's where multiple agents won't do you any good. These are generally specialist cases:

• Your Build Agent doubles as a "Deployment Agent", running a script that installs your product onto the server itself. In a Lab Managament environment such agents are common.
• Your Build Definition requires read or write access to a shared resource (a common fixed file directory for example). Multiple agents would battle over this and could cause file locks that break builds.
• Your server is low on resources in general (either memory, or disk IO) and can't handle multiple builds at the same time.
• Your network bandwidth is very low, running multiple builds in parallel could cause timeouts and other issues when downloading sources or saving build results to the server or a file share. Most of these items can be solved by more memory, local storage or more bandwidth and limiting the number of agents is probably only temporary.
• When you have only one build definition and it's a Gated build. In that case Team Build will always run the builds serially. If you have multiple Gated Builds with different workspace configurations, then having multiple agents is beneficial.

Answer 3

As with many things, the answer is "is depends".

A look into how we do it: We have up to 4 build agents on machines that simply build solutions, we could probably go higher, we also have agents (same machines) that run PowerShell scripts to do various things, so really lightweight in comparison and doesn't cause a problem with agents running CI builds. Several agents on one box is fine (nothing powerful, dual core is all), I haven't seen any real negative performance caused by a server running a couple of builds at the same time.

However, with other things like database builds, we only have one agent on a server, otherwise we get locks and various other issues with restoring prior to build/deploy.

If you're maxing out or locking any resources, then you may want to stick with one build agent per server, however I would say "give it a go" and see, you might find that it's better to have a slightly longer build time when you have a queue of builds lined up, but it chews through your build list faster.