Is there really any difference between NUnit, MSTest, etc.?

Question

Is there really any difference between NUnit, MSTest, etc.?

For example, can any of them detect what code changed since the last build and only run the affected unit tests?

Do any of them have tight integration into database setup/rollback for integration-style tests?

Do any of them allow for scripting TCP ports or web services to test the communication layer?

Answer 1

Detecting what code changed since the last build and running only the affected unit tests isn't really the responsibility of the unit testing framework but rather the responsibility of the build software/scripts.

As far as a unit testing framework goes, they are all fairly comparable to each other, although MSTest is the youngest (I believe). As far as I know, none of them have any implicit (built-in) integration to databases at all, although they all support some notion of setup and teardown for test classes and methods.

Any of them will provide the ability to call web services, etc. but as far as scripting things like TCP ports that is best left to isolation or mocking frameworks, not unit testing frameworks.

Answer 2

Detecting what code changed since the last build and only running the affected unit tests is part of what Continuous Integration (CI) servers can do for you. James Black is quite correct that you want to run the relevant suite of tests when you make a change, not just one test or one fixture. However, you may have a shared library project with a suite of tests and a dependent application project with its own suite of tests. Changes to the application code aren't going to affect the library code, so you can set up your CI build to run all the tests if the library code changes but only the application tests if the application code changes. (You may still want to do nightly builds.) This also depends on your revision/source control system; it may not be possible depending on the combination of source control and CI software.

Matthew Scharley is also correct in that dependencies are best handled by isolation/mocking frameworks for the purpose of unit tests. Both NUnit and MSTest (as well as most of the other xUnit frameworks) can also be used for full integration tests (ones that use the actual dependencies, such as calling web services).

For a very biased view on NUnit vs. MSTest, see MSBuild, NAnt, NUnit, MSTest, and frustration. For a strict comparison (somewhat biased in the other direction), see Comparing the MSTest and Nunit Frameworks.

From a personal standpoint (pure opinion), the key features I need have been available in NUnit, so I've seen no reason to evaluate MSTest. Visual Studio Team System has very tight integration with MSTest (I saw a nice demo that showed code coverage visually); if my company were using that (very expensive) version of VS, MSTest might be worth considering. One key question would be if we could get it working with our CI server.

Answer 3

You can use ant to help run only the tests for the code that last changed, but, that defeats the purpose of unit tests, as you may have broken something that isn't in the file you changed, so all the tests should be ran.

You can design tests to test the database or webservice layers, how much you can do is based on your own ability.

There are problems with the design of unit tests, but that is a problem common to many systems, pattern density.

If you want to do a different style of unit tests you can use the pattern density link above to look at a possible solution, using AOP. You could then write your own layer for using scripts for tests.

MS has a database unit test framework that uses TSQL, but it doesn't automatically do rollbacks at the end of the test, but the tests are scripted out.