CODEWITHSUNDEEP
unit-testing
Wouter Schut
Wouter Schut

Reputation: 926

Verify Unit test by breaking Code on purpose

I want to test (proof) whether my Unit tests actually tests everything it needs to. Specifically how do I check whether I didn't miss certain asserts?

Take for instance this code:

int AddPositives(int a, int b)
{
    if (a > 0 && b > 0)
        return a + b;
    return -1;
}

And someone wrote a Unit test like so:

    [Test]
    public void TestAddPositives()
    {
        Assert.AreEqual(3, AddPositives(1, 2));

        AddPositives(0, 1);
    }

Clearly an assert was missed here, which you might catch in a code-review. But how would you catch this automatically?

So is there something which breaks tested code on purpose to detect missing Asserts? Something which inspects the bytecode and changes constants and deletes code to check whether things can be changed without the Unit test failing.

Upvotes: 0

Views: 301

Answers (2)

Dirk Herrmann
Dirk Herrmann

Reputation: 5939

There are several approaches that can help avoid the problem you have described:

1) The approach you mention (to 'break' the code) is known as mutation testing: Create 'mutants' of the system under test and see how many of the mutants are detected by the test suite. A mutant is a modification of the SUT, for example, by replacing operators in the code: One + in the code could be replaced by a - or a *. But, there are many more possibilities to create mutants. The English Wikipedia has an article about mutation testing. There you also find a number of references, some of which list tools to support mutation testing.

Mutation testing may help you detect 'inactive' test cases, but only if you have some reference that indicates, which mutations should have been detected.

2) Test-first approaches / test-driven development (TDD) also helps to avoid the problem you have described: In a test-first scenario, you write the test before you write the code that makes the test succeed. Therefore, after writing the test, the test suite should fail because of the new test.

Your scenario, namely that you forget to add an assertion, would be detected, because after adding your (not yet complete) test your test suite would not fail, but rather continue to suceed.

However, after the code is implemented, usually additional tests are implemented, for example to also address boundary cases. In these cases, the code is already there and you would then have to temporarily 'break' it to also see the additional tests fail.

3) As was already pointed out by others, coverage analysis can help you to detect the lack of tests that cover a specific part of the code. There are different types of coverage, like statement coverage, branch coverage, etc. But, with a good quality test suite, a piece of code is often covered many times to address boundary cases and other scenarios of interest. Then, leaving out one test case may still not be detected.

Summarized, while all these approaches can help you somehow, none of them is bullet proof. Neither is a review, because also reviewers miss some points. A review may, however, bring additional benefits, like, suggestions to improve the set of tests or the test code.

Upvotes: 1

mark_h
mark_h

Reputation: 5477

Some code coverage tools such as NCrunch (excellent but not free) will annotate your code lines to show whether a test hits them.

In the example you gave NCrunch would show a small black dot next to the "return -1;" line. This would indicate that no existing test passes through that line of code and therefore it is untested.

This is not perfect however since you could still write a test that hit that line of code without asserting that it returned -1, so you can't assume that just because you have 100% coverage that you have written all the meaningful tests. So it can tell you that return -1 is definitely not unit-tested but it would not tell you that you had failed to test a boundary condition (such as checking what happens when a = 0)

Upvotes: 0

Related Questions