CODEWITHSUNDEEP
unit-testingparallel-processingnunit
Daniel Powell
Daniel Powell

Reputation: 8283

Nunit parallel lifecycle and parallel test failing sometimes

I am having an issue with tests sporadically failing when using nunit 3 and using parallel test running.

We have a number of tests that currently are structured as follows

[TestFixture]
public class CalculateShipFromStoreShippingCost
{
    private IService_service;
    private IClient _client;

    [SetUp]
    public void SetUp()
    {
        _service = Substitute.For<IService>();
        _client = new Client(_service);
    }  

    [Test]
    public async Task WhenScenario1()
    {
        _service.Apply(Args.Any<int>).Returns(1);
        var result = _client.DoTheThing();
        Assert.IsTrue(1,result);
    }

    [Test]
    public async Task WhenScenario2()
    {
        _service.Apply(Args.Any<int>).Returns(2);
        var result = _client.DoTheThing();
        Assert.IsTrue(2,result);
    }
}

Sometimes the test fail as one of the substitutes is returning the value for the other test.

How should this test be structured so that with Nunit it will run reliably when done in parallel

Upvotes: 2

Views: 1284

Answers (1)

Charlie
Charlie

Reputation: 13681

You haven't shown any Parallelizable attributes in your example, so I assume you are using the attribute at a higher level, most likely on the assembly. Otherwise, no parallel execution would occur. Further, since you say the test cases are running in parallel, you have apparently specified ParallelScope.Children.

The two test cases shown in your fixture cannot run in parallel. You should bear in mind that the SetUp method runs for each of the tests. So each of your two tests sets the value of _service, which is part of the state of the single instance of CalculateShipFromStoreShippingCost, which is shared by both tests. That is why you are seeing the "wrong" substitute being returned at times.

It is not possible for two test cases to run reliably in parallel if they both change the state of the fixture. Note that it does not matter whether the assignment to _service takes place in the test method itself or in the SetUp method - both are executed as part of the test case. So, you have to either stop running these two cases in parallel or stop changing the state.

To stop running the tests in parallel, you simply add [NonParallelizable] to each test method. If you are not using the latest framework version, use [Parallelizable(ParallelScope.None)] instead. Your other tests will continue to run in parallel, but these two will not.

Alternatively, use ParallelScope.Fixture at the assembly level. This will cause fixtures to run in parallel by default, while the individual test cases within them each run sequentially. When using ParallelizableAttribute at the assembly level, it is sometimes best to take a more conservative approach, adding in more parallelism within some fixtures where it is useful.

An entirely different approach is to make your tests stateless. Eliminate the _service member and use a local value within the test method itself. Each of your tests would add two lines like...

var service = SubstituteFor<IService>();
var client = new Client(service);

As shown in your example, I would imagine you are getting very little performance gain from running the two methods in parallel, so I would not use that last approach unless I saw a specific performance reason to do so.

As a final note... If you use make your fixtures run in parallel by default (either with an assembly-level attribute or with attributes on each fixture) and place no Parallelizable attribute on your test cases, NUnit uses an optimization, whereby all the tests within the fixture run on the same thread. This saving in context changes will often make up for the loss of any performance improvement you hoped to get through running in parallel.

Upvotes: 5

Related Questions