Team Build now Painfully Slow

Question

We're running into performance issues with our implementation of Team Foundation Build Server and I'm running out of ideas on how to speed things up. We've already added a few PropertyGroup elements to increase the performance on several steps (SkipClean, SkipLabel, SkipInitializeWorkspace), but I think we need to undergo a major restructuring to fix things. Here's our setup:

• We've got about 40 web applications that are each very different, but run off a bunch of shared assemblies
• Each of these web applications has their own solution;
• There are around 10 to 25 shared assemblies referenced by each of these web applications;
• There exists a build definition containing all the solutions which is fired on each check-in to the trunk;

And here's the basic problems we're encountering

• During the build, it will build each shared assembly as many times as it is referenced, rather than building once and using for each app
• File copy time is reeeeally slow for the drop directory. It has to be over network share and won't take a local path.
• Every so many builds, one or more of the output files gets "locked" and causes the build to break even if compilation is fine.
• And another thing - I've also tried separate build definitions, but doing so will also force another workspace to be gotten on Get Latest version. I'd much rather have it be that the build server contains one version of the trunk to build off of.

Over the last several months we've given in to lethargy and ignored this problem, but now the build time is over an hour to an hour and a half.

I'm toying around with the idea of learning and switching to Cruise Control for the greater control I'd have. Anyone disagree with that?

Any help is most appreciated. Thanks!

Answer 1

So here's what I've done, and I've gotten the build down to 9 minutes. For the amount of projects I'm compiling, I'm fine with that.

• Created a solution which contains all shared libraries and all webs. There was a lot of extra work in this step because I had to fix a bunch of references that were legacy code or otherwise stored in multiple places.
• The thing that ended up saving the most time was to perform a file system move rather than a network copy for all output files. Since our drop location actually sits on the build server, it makes sense.

To perform the move, I just override the CoreDropBuild target within the TFSBuild.proj file:

<Target Name="CoreDropBuild"
        Condition=" '$(SkipDropBuild)'!='true' and '$(IsDesktopBuild)'!='true' "
        DependsOnTargets="$(CoreDropBuildDependsOn)" >
             <Exec Command="move $(BinariesRoot)\Release d:\BuildOutput\$(BuildNumber)\Release"/>    
</Target>

Answer 2

Do you really need to build everything in every web app? If the shared assemblies haven't changed, why build them over and over again?

Here's a thought:

1. Let every web app have their own \lib folder.
2. Place every shared assembly in the lib folder.
3. Let the web app only reference shared assemblies from its local lib folder.
4. Check everything in.

Now a build shouldn't start unless something have changed, and the build will not include the shared assemblies.

• There should be a central folder containing all shared assemblies.
• Any change here should propagate to all local \lib folders.
• Finally, any shared assembly should be copied to the central folder whenever they change.

The idea here is to have a shared assembly project to only know of the central folder. This will simplify any post-build action necessary to copy the build.

The central folder must be managed in such a way that any changes will be copied to all referencing web app.

Answer 3

Speaking from personal experience on the CruiseControl suggestion - remember it's a continuous integration "framework". It won't solve all your problems out of the box (componentized builds, firing on each component change, and serialized builds though will make things a lot better). It'll take quite some configuration (and maybe even customization) to get things how you want, so be prepared to invest some time. Of course, you'll reap a massive payoff in the long run if your build time gets right down - if you can't ignore the problem any more, it's worth investing some time on a better CI solution.

Be aware though that any CI effort is only as good as the policies you have in place. We had huge policy voids when it came to version labeling, releasing, dependencies, beta releases of binaries, archiving builds... and many other issues that we didn't even consider at the time.

Also, be prepared to dedicate at least some resources to maintaining the thing. It's not a full-time job (and I for one love doing it, since it produces continual process improvement). Our customizations have taken us from a 2 hour monolithic build of our first product to over 400 components in 20 products that build in parallel on multiple machines within about 20 minutes, so it's well worth it.

Answer 4

First, it sounds as if all of your web apps are contained within the same Team Project. If that's true, split them out into logical groupings. Typically a single Team Project should comprise of a single Deployment model.

Second, split the shared assemblies into a their own Team Project. Once moved you have a couple choices, you can branch either the source or the compiled DLLs to the Team Projects that need them. They can have their own unit tests and you can extend team build to automatically merge on a successful test if you are so inclined.

To sum up, you need to simplify your build strategy.