CODEWITHSUNDEEP
unit-testingnunit
JamesFaix
JamesFaix

Reputation: 8645

NUnit - Can I group tests by a key and run tests in each group in parallel, but each group in series?

I have a large suite of tests that are currently running in series. I would like to make the suite parallelizable as much as possible. One big problem I have is that some tests require certain global application state to be set one way, and some require it to be set another way. Is there a way I can make the NUnit test runner group my tests based on which global state they require, and then run the tests within each group in parallel with each other, and modify the global state between groups?

For example, let's say there is a global setting Foo and a custom attribute [RequiresFoo(X)] that can be used to annotate which Foo value a test requires. At runtime I want NUnit to group all tests by their argument to RequiresFoo, counting unmarked tests as having some default Foo value.

Then for each group, I want it to

  1. Set Foo = N where N is the Foo value for that group.
  2. Run all tests in that group in parallel.

In an ideal world I would have a mock system for this global state, but that would take a lot of time that I don't have right now. Can I get NUnit to do this or something like it?

Note, I need to be able to execute any method between groups, not just set a variable. The global context I'm actually dealing with can involve starting or stopping microservices, updating configuration files, etc. I can serialize any of these requirements to a string to pass to a custom attribute, but at runtime I need to be able to run arbitrary code to parse the requirements and reconfigure the environment.

Here is a pseudo-code example.

By default tests execute in series like this:

foreach (var t in allTests)
{
    Run(t);
}

NUnits basic parallel behavior is like this:

Parallel.ForEach(allTests, t => Run(t));

I want something like this:

var grouped = allTests
    .GroupBy(t => GetRequirementsFromCustomAttributes(t));

foreach (var g in grouped)
{
    SetGlobalState(g.Key);
    Parallel.ForEach(g, t => Run(t));
}

Upvotes: 2

Views: 1413

Answers (1)

Charlie
Charlie

Reputation: 13681

As you state the problem, it's not yet possible in NUnit. There is a feature planned but not yet implemented that would allow arbitrary grouping of tests that may not run together.

Your workaround is to make each "group" a test, since NUnit only allows specification of parallelization on tests. Note that by test, we mean either a test case or a group of tests, i.e. fixture or namespace suite.

Putting [Parallelizable] on a test anywhere in the hierarchy causes that test to run in parallel with other tests at the same level. Putting [NonParallelizable] on it causes that same test to run in isolation.

Let's say you have five fixtures that require a value of Foo. You would make each of those fixtures non-parallelizable. In that way, none of them could run at the same time and interfere with the others.

If you want to allow those fixtures to run in parallel with other non-Foo fixtures, simply put them all in the same namespace, like Tests.FooRequred. Create a SetupFixture in that namespace - possibly a dummy without any actual setup action. Put the [Parallelizable] attribute on it.

The group of all foo tests would then run in parallel with other tests while the individual fixtures would not run in parallel with one another.

Upvotes: 1

Related Questions