Running integration tests across multiple machines

Question

I have an n-tier application with the different tiers running on different machines. I need to execute coded integration tests on those different machines, preferably using the MSTest framework since that's what everything else is written in.

There's an orchestration problem, in that before the tests on machine "B" can run, machine "A" needs to be set up correctly. VS out of the box doesn't seem to handle that scenario so I need something else.

On the surface TFS lab manager would seem to be what I need to manage the machines, deploy the code and run the appropriate tests but the wrinkle is that the code doesn't live there - we use Perforce for source control.

So

a) Has anyone successfully used TFS lab manager for test deployment using code that came out of another source control system? If so, how did you set it up?

or

b) Are there any alternative testing frameworks that will let me deploy code to multiple machines, execute coded tests and gather success/failure results?

Answer 1

I'm going to answer my own question with what I actually did: abandoned the lab idea as being just too much work for the job at hand!

It looked promising at first and I'm sure it is possible with sufficient effort but as the other answers noted it would mean substantial edits to the lab workflow to achieve my goal - way more than I can justify. Not only was there a problem with the code not being built with TFS but I also ran into problems trying to tell lab/test manager how to run which tests where.

In the end I cobbled a halfway house solution together using a powershell script to set up the environment (revert hyper-v snapshots, copy a few files around and run some remote installs) and the standard MSTest framework built into VS for executing tests against the environment.

Answer 2

Based on your comments

The code I'm testing is built locally using VS. The integration tests are manually kicked off.

I would expect any CI platform to operate with a labelled build as input.
Actually on your (both) questions: I think that until you invest the effort to upgrade to a stable Build environment you are doomed to do this by hand/by self-brewed scripts.

Having a build server is a really good thing and, if you are into setting up a modern development environment, you basically can't do without it.

If you 're attracted by TFS Lab Manager, going with TFS-Build is a must. If you go down this path, you should consider toggling everything into TFS - including source control.

I 'd say that using Perforce as source control & TFS for Build/Test is absolutely possible, yet it can be a headache to set up the build properly (literally everything that comes out of the box regarding source control needs to be refactored to p4-friendly calls).

Answer 3

a) I haven't done this with different source control, but I think it's definitely possible with TFS Lab Manager. The TFS Workflow & Test Controller/Agent model works based on Windows Workflow in Lab manager, so you would want to modify the workflow for the build (using the XAML editor or Workflow UI in VS) and replace any TFS source control calls to your own task set to fetch/version/checkin-out code. In fact the TFS Lab workflow is divorced from the build workflow and can just pick up built binaries from a UNC path or something. If you chose the latter, you could build the code with MSBuild and some other driver like Jenkins and then use TFS Lab Manager to execute the tests on each tier correctly.

b) Conversely, you could use Jenkins entirely and the VMWare plug-ins to spin up and down VMs for testing. You could use the Powershell plug-in and powershell remoting or some other MSTest plugin to execute the remote tests. This option seems harder to me, but I think both could work.