CODEWITHSUNDEEP
c#unit-testingnunit
mmattax
mmattax

Reputation: 27710

NUnit: Running multiple assertions in a single test

I have been asked to write a testing application that needs to test a new stored procedure on multiple rows in a database, in essence I want to do something like this:

[Test]
public void TestSelect()
{
    foreach(id in ids)
    {
        DataTable old = Database.call("old_stored_proc",id);
        DataTable new_ = Database.call("new_stored_proc",id);

        Assert.AreEqual(old.Rows[0]["column"],ne_.Rows[0]["column"]);
    }
}

When I run this test, if 1 row doesn't match the other, the entire test fails; instead I would like to count how many times the assertion was passed and how many times it has failed. Is there a way to do this with NUnit?

I realize that NUnit might be overkill and this is a simple task without it...I just wanted to learn it. ;)

Upvotes: 7

Views: 7285

Answers (8)

yoyo
yoyo

Reputation: 8726

I recently had the same issue. I combined the idea of counting errors with Yann Trevin's mention of Assert.Multiple into an extension method for IEnumerable<T> that lets me do things like:

[Test]
public void TestEvenNumbers()
{
    int[] numbers = new int[] { 2, 4, 12, 22, 13, 42 };
    numbers.AssertAll((num) => Assert.That((num % 2) == 0, "{0} is an odd number", num));
}

Which results in the NUnit output:

TestEvenNumbers:
  5 of 6 tests passed; 0 inconclusive
FAILED: 13:   13 is an odd number
  Expected: True
  But was:  False

  Expected: 6
  But was:  5

And the solution to the OP's problem would be:

[Test]
public void TestSelect()
{
    ids.AssertAll(CheckStoredProcedures);
}

private void CheckStoredProcedures(Id id)
{
    DataTable old = Database.call("old_stored_proc",id);
    DataTable new_ = Database.call("new_stored_proc",id);

    Assert.AreEqual(old.Rows[0]["column"], new_.Rows[0]["column"]);
}

Here is the extension method (note that I used "All" instead of "Multiple" for consistency with Linq terminology):

using System;
using System.Text;
using System.Collections.Generic;
using NUnit.Framework;

public static class NUnitExtensions
{
    public static void AssertAll<T>(this IEnumerable<T> objects, Action<T> test)
    {
        int total = 0;
        int passed = 0;
        int failed = 0;
        int inconclusive = 0;
        var sb = new StringBuilder();

        foreach (var obj in objects)
        {
            total++;

            try
            {
                test(obj);
                passed++;
            }
            catch (InconclusiveException assertion)
            {
                inconclusive++;
                string message = string.Format("INCONCLUSIVE: {0}: {1}", obj.ToString(), assertion.Message);
                Console.WriteLine(message);
                sb.AppendLine(message);
            }
            catch (AssertionException assertion)
            {
                failed++;
                string message = string.Format("FAILED: {0}: {1}", obj.ToString(), assertion.Message);
                Console.WriteLine(message);
                sb.AppendLine(message);
            }
        }

        if (passed != total)
        {
            string details = sb.ToString();
            string message = string.Format("{0} of {1} tests passed; {2} inconclusive\n{3}", passed, total, inconclusive, details);

            if (failed == 0)
            {
                Assert.Inconclusive(message);
            }
            else
            {
                Assert.AreEqual(total, passed, message);
            }
        }
    }
}

Upvotes: 0

Ray
Ray

Reputation: 192396

You can use [TestCase()] attribute if a simple hard coded list of IDs.

[Test]
[TestCase(1234)]
[TestCase(5678)]
[TestCase(7654)]
public void TestSelect(int id)
{
    DataTable old = Database.call("old_stored_proc", id);
    DataTable new_ = Database.call("new_stored_proc", id);

    Assert.AreEqual(old.Rows[0]["column"], new_.Rows[0]["column"]);
}

This will generate three separate tests for each ID and whatever nunit test runner you use will display pass/fail counts.

If need to generate a dynamic list of IDs then recommend using [TestCaseSource()] attribute.

Upvotes: 1

Yann Trevin
Yann Trevin

Reputation: 3823

I know that the question is specifically about NUnit, but interestingly enough, Gallio/MbUnit has a feature which allows to run and catch several assertions at once.

[Test]
public void MultipleTest()
{
    Assert.Multiple(() =>
    {
       Assert.IsTrue(blabla);
       Assert.AreEqual(pik, pok);
       // etc.
    }
}

The Assert.Multiple is catching all the failing assertions and is going to report them at the end of the test.

Upvotes: 5

Scott Lawrence
Scott Lawrence

Reputation: 7253

Based on the objective you laid out, the entire test should fail if one row doesn't match another. Counting the number of times an assertion passes or fails gives you less information than a comparison of the outcome you expected with the outcome you actually got.

Upvotes: 0

Mark Roddy
Mark Roddy

Reputation: 27986

1) If the id's are constant and not looked up at test run time, create a separate unit test fixture for each id. That way you will know which id's are actually failing. See here for a write up on the problems with data driven tests:
http://googletesting.blogspot.com/2008/09/tott-data-driven-traps.html

2) If you need to dynamically look up the id's making it impossible to create a fixture for each id, use akmad's suggestion with one change. Keep a list of id's where the values are not equal and add the list to the error message. It will be extremely difficult to diagnose a failing test that only states the number of errors, as you won't know what id's cause the errors.

3) I don't know how difficult it would be to do in NUnit, but in PyUnit, when we need to run tests on dynamically generated data, we dynamically create tests fixtures and attach them to the TestCase class so that we have a failed test for each piece of data that does not pass. Though I imagine this would be much more difficult without python's dynamic abilities.

Upvotes: 5

akmad
akmad

Reputation: 19791

Seems like you are just Asserting the wrong thing. If you want to check all the values and then assert that there are no errors (or show the number of errors) then try this:

[Test]
public void TestSelect()
{
    int errors = 0;
    foreach(id in ids)
    {
        DataTable old = Database.call("old_stored_proc",id);
        DataTable new_ = Database.call("new_stored_proc",id);

        if (old.Rows[0]["column"] != new_.Rows[0]["column"])
        {
            errors++;
        }            
    }

    Assert.AreEqual(0, errors, "There were " + errors + " errors.");
}

Upvotes: 9

Ilya Kochetov
Ilya Kochetov

Reputation: 18463

I would count the number of rows which do not match and then would write an assertion which will compare this number with 0 and would return the number of non matching strings in the message.

you could also use Assert.Greater for this.

P.S. In principal you should try to do one assertion per unit test. That's the gist of it.

Upvotes: 1

DKATDT
DKATDT

Reputation: 908

Well you could declare a counter and then assert the value of the counter to determine pass/fail

Also, you could do the bulk of the work in the test setup, and then just create multiple tests.

I'm not clear as to why you need all the assert stmts in the same test.

Upvotes: 0

Related Questions