MSBUILD (VS2010) very slow on some machines

Question

I've got a very weird one.

I have about a dozen PC's in our development department that all suffer the same issue: commandline builds using msbuild4.0 (VS2010) are way slower than should be. 4x to 5x slower as expected.

All machines are HP z400 Workstations (quad-core Xeon+hyperthreading, 2.4 GHz, 6GB RAM) running Windows 7 Pro 64bit, or HP Elitebook notebooks (Core-i7 4GB RAM) also on Win7 X64. If I take one of those with the vanilla factory pre-installed Win7, install VS2010 and do a build they are as fast as expected. If I install them with our company standard software image, they became 4x to 5x slower on the same VS project.

The same company software image on Lenovo laptops (Core-i5) or desktops (Core-i7) shows no measurable difference between the company image and the factory pre-installed Win7. It's even stranger: If I install VirtualBox with a Win7 image on a HP system with the problem the virtual machine DOESN'T have the problem.

Every benchmark tool I have tried shows no measurable difference between the company image and the pre-installed Win7. Only msbuild is affected and only on the HP machines running Win7 on the company image.

Before you ask: I have disabled all software/background processes in the company image, but that doesn't make any difference. It's obviously not something running in the background that somehow interacts with msbuild. My best guess is that on the HP hardware some settings gets changed that affects msbuild. This doesn't happen on other hardware. (And the VS2010 GUI is not used/executed at all. I am aware this can interact with the msbuild if both try top access the same solution/files. Antivirus doesn't make any difference either.)

Anybody have any idea what could possible influence msbuild to slow down? Any suggestion, no matter how far-fetched, is welcome.

Answer 1

Answering this myself as I have found the problem. The computers in question all have the "Entrust Entelligent Security Provider" software installed. (A certificate manager our company uses for encrypted mail handling.)

.NET4 (in which VS2010 is written) and the compilation of .NET4 code require a large amount of certificate checks and this software (which get's called in the process) apparently has issues on multi-core PC's. The more cores how slower things become.

The core-i5 systems have the problem as well, but not that badly. They are still usable. The i7 and Xeon system (8 cores) slow down to the point of becoming nearly unusable.

Unfortunately I can't get rid of this Entrust software. It's part of the company mandated software suite. I just hope the supplier has a solution for this.

EDIT: Problem is solved with version 10 of Entrust Entelligence. It does not set itself by default as first certificate handler in the system anymore.
On v9 a manual edit of the registry is required to put the MS handler back in default position.
That was the real problem: Every certificate check went first through the Entrust DLL which pushed everything in a queue which got emptied by only a single-thread/CPU. Then it discovered it wasn't supposed to deal with this certificate and handed it of to the Microsoft handler.

Answer 2

A diagnostic report will show how MSBuild is working at a detailed level, but won't always help diagnose performance issues. Try running a normal build adding the following additional options to the msbuild command line:

/ds /clp:performancesummary

This will output a diagnostic summary about node usage, and also add a detailed performance summary to the end of the console output of the build, showing how long each task is running. If you use a file logger you can add the ";performancesummary" option to the end of your file logger parameters (/flp:...).

One other thing to keep an eye on would be to reduce the AssemblySearchPaths property to use only the paths you are using in your project, it has been a cause of performance issues on some builds I've seen. Just search in Microsoft.Common.targets, redefine it in a standard import used by all your projects (you do have one of those, right:) and redefine it prior to where the standard files get imported, removing any of the {...} items that you don't make use of.

Answer 3

Run MSBuild from the command line with the verbosity set to diag:

MSBuild mysolution.sln /v:diag

That will produce a list containing the amount of time spent per task. That should give you some information to go on.

Answer 4

Firstly I assume you are not doing anything daft like building on a network location or mapped drive, or redirected "Documents" folder?

Next step is to use Task Manager to see which processes are slowing you down. Is processor usage at 100%? What processes are active? Is it paging a lot (look at "page faults delta")? Which processes?

I have to say this is sounds like antivirus. You should exclude OBJ, LIB, ILK, PDB and similar file types from antivirus on-access scanning.

I know you said "Antivirus doesn't make any difference" but you haven't confirmed that you have tried turning it off altogether.

Also look at search indexer, continuous backup or similar processes which might be responding to the build activity.