CODEWITHSUNDEEP
c#visual-studiounit-testingnunitnunit-3.0
Michael Hennigan
Michael Hennigan

Reputation: 375

Unit Tests failing when I Run All Tests but pass when I Debug

I'm using NUnit3 in Visual Studio 2017 and doing TDD. Something really strange is happening since I updated my code to make my latest test pass.

Now, 3 of my other tests are failing when I click Run All Tests, as below:

enter image description here

It is telling me that the actual and expected values in my Assert method are not equal.

However, when I put a breakpoint at the line where the Assert method is and start debugging, the stacktrace is showing that expected and actual are the same value and then the test passes, as below:

enter image description here

Am I doing something stupid or could there be a bug in VS2017 or NUnit or something?

This ever happen to anyone else?

[Edit: I should probably add that I have written each test as a separate class]

Upvotes: 19

Views: 20668

Answers (4)

Alex Konnen
Alex Konnen

Reputation: 747

I encountered this phenomenon myself, but found the cause quite easily. More concretely, I tested some matrix calculations, and in my test class I defined data to calculate with as a class variable and performed my calculations with it. My matrix routines, however, modified the original data, so when I used "run tests" on the test class, the first test corrupted the data, and the next test could not succeed.

The sample code below is an attempt to show what I mean.

[TestFixture]
public void MyTestClass()
{
    [Test]
    public void TestMethod1()
    {
        MyMatrix m = new MyMatrix();

        // Method1() modifies the data...
        m.Method1(_data);
    }

    [Test]
    public void TestMethod2()
    {
        MyMatrix m = new MyMatrix();

        // here you test with modified data and, in general, cannot expect success
        m.Method2(_data);
    }

    // the data to test with
    private double[] _data = new double[1, 2, 3, 4]{};
}

Upvotes: 0

Chris
Chris

Reputation: 11

I experienced a similar issue in Visual Studio 2017 using MSTest as the testing framework. Assertions in unit tests were failing when the tests were run but would pass when the unit tests were debugged. This was occurring for a handful of unit tests but not all of them. In addition to the Assertion failures, many of the units tests were also failing due to a System.TypeLoadException (Could not load type from assembly error). I ultimately did the following which solved the problem:

  1. Open the Local.testsettings file in the solution
  2. Go to the "Unit Test" settings
  3. Uncheck the "Use the Load Context for assemblies in the test directory." checkbox

After taking these steps all unit tests started passing when run.

Upvotes: 0

Nkosi
Nkosi

Reputation: 247511

The failing tests share a resource that affects them all when tested together. Recheck the affected tests and their subjects.

You should also look into static fields or properties in the subjects. They tends to cause issues if not used properly when designing your classes.

Upvotes: 15

gturri
gturri

Reputation: 14629

Some subtle differences might occur. For instance if a first test change a state which affects the behavior of a second test, then the outcome of this 2nd test may not be the same if I run it alone.

An idea to help understand a test failure when a breakpoint can't be used, could be to add logging.

Anyway, to answer your questions:

This ever happen to anyone else?

Yes

Am I doing something stupid or could there be a bug in VS2017 or NUnit or something?

I bet that it's neither: just a case a bit more subtle

Upvotes: 1

Related Questions