CODEWITHSUNDEEP
.netunit-testingcode-coveragemstest
Scott Wegner
Scott Wegner

Reputation: 7473

Finding .NET code covered exclusively by a set of tests

For a suite of .NET tests (MSTest), is there a way to find code blocks that are covered exclusively by a particular subset of the tests (i.e. a single TestClass?)

Here's my scenario: I'm working in an old code base which has built up a large and slow suite of unit and integration tests over time. We'd like to reduce the overall runtime of the suite. One approach is reducing redundancy between integration tests and unit tests. In fact, there are likely integration tests that are completely redundant to some unit tests.

Here's what I'd like to do:

  1. Collect code coverage over the full suite of unit and integration tests.
  2. Find integration tests which don't cover any blocks not covered by other tests.
  3. Manually validate that the reported tests are completely redundant, and if so, remove them.

Edit: Based on feedback so far, let me emphasize that code coverage is only a tool and paints an incomplete picture of what tests are really testing. It's important to understand a test before discounting its value.

Our tests are written in MSTest and run using Visual Studio. I'm familiar with collecting code coverage, but I'm not sure how to query through it.

Upvotes: 1

Views: 313

Answers (3)

Shaun Wilde
Shaun Wilde

Reputation: 8358

OpenCover has a cover by test feature which should help you gather the information you want. The XML however may get a bit big depending on your tests and code base. Have a look at the wiki on how to use it.

Upvotes: 1

Daniel Mann
Daniel Mann

Reputation: 58980

You are making a really, really dangerous assumption: That the integration test serves no purpose if other code is already testing it.

This is the classic code coverage pitfall. Code coverage is essentially meaningless. All it can show you reliably is code that isn't even attempted to be tested. It cannot validate that the tests are testing anything worthwhile, or that your software doesn't have bugs in it.

You can't just blindly compare code coverage for unit tests and integration tests and then start removing integration tests when unit tests are already testing the code. Each integration test should be evaluated on its own merit for whether it's testing something valuable that can't be validated with a unit test.

Upvotes: 2

Ira Baxter
Ira Baxter

Reputation: 95324

I'm a little unclear on whether you want to

  • find tests that redundantly cover the same code and remove them (you need to be careful here: you may have two tests on the same code, that test different aspects of the functionality), or
  • whether you want to find all tests that cover a specific, identified bit of code.

In any case, what you need to do this is a test coverage tool that can collect coverage vectors on a per-test basis, and then a way to determine which coverage vectors cover specific code, and/or which vectors intersect.

Our Test Coverage tools can do this for a variety of languages, including C#. You will have to modify the MSTest framework slightly to keep track of which test it is running, and for each test, to dump a test coverage vector for that test when the test is complete.

The Test Coverage Vector Display UI that is part of the tool will let you combine arbitrary sets of vectors, and compare those combinations (intersect? complement? different?) with other combinations. To determine which individual vectors intersect, you'll need to write a small routine to read test coverage vectors (essentially bit strings) and compute their intersections pairwise for all pairs.

Upvotes: 0

Related Questions