CODEWITHSUNDEEP
tfsbuildazure-pipelinesbuild-agentparallel-testing
Noa Sasson
Noa Sasson

Reputation: 23

How not to lose parallel testing with tfs build and keep build agents work

I'm setting up a TFS testing build and want to support full parallel testing. Only some of our tests run in parallel and therefore after they finished running on the build agent - the build agent do not getting new tests to test. Therefore, a build that took us 30 minutes start to take an hour and more because all the renaming tests running on only one build agent. All the other 20 build agents stop to run after several minutes because they finished running their bulk of tests. We run our parallel tests with MSTest2. Our RunSettings.runSettings declare to run as parallel and are correct.

We tried different Advanced Execution Options on the build. We tried based on past running time of tests. based on number of tests and agents and based on test assemblies. None of them work - after the agent finished his bulk of tests he just go back to idle. We tried giving the machine in the build to get only one test each bulk and it was indeed faster but we lost the privilege of parallel testing on each machine.

We want our build to run on this way: We start it, and the tests start to run. All the machines will get count of tests to run. When the build agent finish to run all the tests he was given, it will get tests from a machine that didn't finish to run his tests. This way we'll enjoy parallel testing build and parallel tests on a machine. We don't want to get to the point where our build has 2 steps - 1 step that run the tests parallel and another one that run our tests which are not parallel.

Does it possible to create a build that answer all of the above? We started to lose our heads because of this and we didn't find examples to builds that will do as we want. Or how at least change the settings to prevent from build agent to stop getting new tests after he finished his bulk.

Best regards and I hope that I made myself clear enough.

Upvotes: 1

Views: 154

Answers (1)

jessehouwing
jessehouwing

Reputation: 114661

No this is not possible. The tests are distributed at the start of the test run. What you can do is the following:

  • Make sure all your slow tests are marked using a [TestCategory("slow")] attribute.
  • Create one Run Tests task in Azure Pipelines and use the filter to run all tests except slow tests. This will distribute the remaining tests across all agents.
  • Create one Run Tests task in Azure Pipelines and use the filter to only run the slow tests. This will distribute all slow tests to all agents

You end up with two test runs, but your tests will be more equally distributed.

As an added advantage your slow tests can optionally be skipped if one of your fast tests already fail.

Basically what I blogged here:

Upvotes: 0

Related Questions