What advantage have other testing libraries over the one provided by Visual Studio 2008?

Question

Or in another way, is there any reason to install NUnit or any other library and use it instead of the one that comes with Visual Studio 2008?

Answer 1

A couple of notes RE MSTest...

1) There is an entire team dedicated to brining features to market (it's literally their full time job). These guys end up having resources for writing documents, integration points into other Microsoft products, discussing advances from the general test community and how these get into Microsoft test products and in general an overall, fully functioning product unit dedicated to the effort. When you go with MSTest, you get the full benefit of these peoples efforts and the dollars Ms invests.

2) Web Testing and Stress Testing. There is yet a WHOLE other dedicated group (separate from the MSTest group) that works on Web testing and stress testing. These are huge additions to the Test stack from Microsoft. These offer ways to simulate online users, use complex bandwidth manipulation, check server performance and much much much more. The Web and Stress testing stuff is a BIG reason to really give the MS Test stack a hard look.

3) Functional testing ... Ms is adding full UI automated testing support and this will really round out their offering. It was missing in terms of earlier versions of MS Test tools, but it must be considered that a fully integrated set of UI testing tools will really amp up the overall stock of an investment in MS test technology.

4) As noted above - Impact Testing. From a developers point of view - this is WAY cool. As mentioned - the tools select the right unit tests passed on code coverage of the unit test. So a dev edits and existing class, etc... and the tool specifically points to the tests that impact the checkin. BTW - this is a GREAT example of the research you gain from that I mentioned in #1.

.... BTW - Full disclosure - I worked on VSTS directly - and totally consider it a part of my family - so yes - I am utterly compromised in terms of favoritism.

As for Non-MS test tools ...

A) You can't disregard the efforts of other companies nor the entire open source community. Really MS still loses if you consider on one hand the entire open source community and all of other company efforts VS several Microsoft product teams. HOWEVER, the MS efforts are mostly coordinated (though not always) and do tend to have at least soft points of integration (though again, not always).

B) Faster turn-around from other vendors / the open source community. Ms is improving - BUT - no - other communities and vendors DO out pace MS in getting things to market. This is not different from test tool innovations.

C) ***** Other testing tools will do a GREAT job with non-ms technology stack stuff. Ms tools will be 'ok' in some cases - and in other cases utterly miss the boat. Many times the big ticket wins from MS tools aren't there for other technology (read NON-MS) technology stacks - ... Other vendors and the open source community (obviously) do provide much more compelling support.

In the end... I'd suggest this: If you're on the MS stack AND have an investment in Visual Studio - particuarly with MSDN - and thus have a major amount of access and investment in buying MS tools .... REALLY, using SOME FORM of MS development and test tools is a no brainer. I didn't even discuss the test tools integration and usage of the rest of VSTS - holy cow.

However - if you're using other stacks, don't have the end-to-end investment in the MS stack / MS provided developer tools ... If you're operating in a shop where other OS's and such are huge deals for desktop or server installs, etc... Then perhaps minor elements of the MS tool set might be worth checking out - but obviously starting in the MS test tools direction will be somewhat more difficult / costly.

Hope this helps.

Answer 2

We have been using MS test tools since VS2005, and I did look at NUnit as a replacement at one point because of some problems we had with the MS tools. Here are some of my findings:

MS test tools pros

• Easy to convince management to use MS supplied tools (of course this is not a technical issue, but it is probably relevant in a lot of companies)
• Test libraries, test runner and code coverage are all nicely integrated with VS
• Central reporting and other features in a Team Foundation setup
• More or less the same feature set as with NUnit (I get the feeling that NUnit may be a tad better than MS test tools, but the differences are not huge)

MS test tools cons

• VS2005 - really slow, VS2008 a lot better, but still sluggish for large solutions
• Test host is STA which makes it harder to test multi-threaded code
• VS support is a bit clunky because they support test-first and test-after in the same interface (so there are two "new unit test" commands -> more support work)

In the end switching to NUnit was not attractive to us for a couple of reasons:

• Code changes, not a lot but with a large number of tests it still amounts to a bit of work
• Increased cost and complexity per developer seat: if you swap MS Test with NUnit, you also loose the test runner and code coverage. Both can be obtained through other tools such as R#, TestDriven.NET, NCover etc, but it all adds to both the cost and the complexity.

Answer 3

Some advantages:

• (Jon's) You can run your tests in VS 2008 Express
• Lot's of cutting edge features get built on top of the free libraries. Eg. BDD
• In general there is rich extensability when it comes to the OSS testing frameworks.
• You can debug into the source of the OSS frameworks, and make changes if required.
• Arguably, some test frameworks like xUnit are easier to learn and enforce a cleaner side-effect free testing pattern.

Given the awesome testdriven.net supports pretty much all of the OSS testing frameworks, you get the same (if not better) IDE integration that you get with VS 2008.

Answer 4

I have just started to use Visual Studio test projects. I have no experience of using thrid party test products, but I have found that the Visual Studio one appears to be perfectly adequate for most points.

I think that it would boild down to what you have experience in.

If you already know how to use NUnit and/or Rhino mocks then go with them.

If you have no experience of either or aren't too bothered about gaining experience in them then go with the Visual Studio supplied one.

A useful pointer is that the Visual Studio one will create test method stubs and accessors automatically for you if you right click and select "Create Unit Test" from anywhere in your code. Particularly useful if you're not following a strict TDD of writing tests before coding.

Answer 5

NUnit allows you to run tests in a defined order whereas the Visual Studio test framework runs the tests concurrently.

Answer 6

I think most test projects created before VS2008 will nowadays still use other libraries.

For newer projects it comes down to personal taste I guess, or company policies.

Also, not all visual studio versions support the testing framework I think...

Answer 7

Do you want developers who don't have Visual Studio 2008, or who only have the Express edition, to run the unit tests? This is particularly relevant for Open Source projects. If so, I'd go with one of the open source unit test frameworks.