CODEWITHSUNDEEP
.netunit-testingintegration-testing
Brent Arias
Brent Arias

Reputation: 30175

Shoe-horning Integration and System tests into a Unit Test Framework

As shown in a domain-manager article, I am interested in creating an integration test harness that creates a server and many clients, where all of them are running within a single process. But once I achieve this goal I will be very tempted to execute such integration & system tests from within a unit testing framework (NUnit or VS Team test).

What are the pros and cons of choosing to run integration or system tests from within a unit test framework? Here is my own stab:

Pros

Cons

Either way, if my new integration testing code is not placed into a unit testing framework, where would you recommend it be placed? Said differently, what "integration test" frameworks would you recommend?

Upvotes: 3

Views: 467

Answers (3)

Jack Ukleja
Jack Ukleja

Reputation: 13511

If you think about it, most unit test frameworks are not in any way specific to unit testing. They provide tools to run arbitrary bits of code and to validate their outputs. All of this is necessary, if not sufficient, for integration and system testing.

For this reason I would say it's perfectly reasonable for use a "unit' test framework for non-unit testing.

Pros

  • Easy to integrate non-unit tests into your build process (assuming you already have unit tests)
  • Ability to leverage tooling from the unit test framework ecosystem (GUIs etc)
  • Test language will be familiar to devs (typically will use same general purpose language as used in development)
  • Unit test frameworks often have many features good for non-unit testing e.g. timeouts, categories.

Cons

  • No assistance with high level tasks such as starting or monitoring process, installing, or database setup often required for non-unit tests.
  • Because they use a general purpose programming language, tests may become verbose due to the complexity in setting up tests, particularly system tests
  • Most unit tests frameworks don't provide any high level tools such as recording/playback of test plans, or UIs suitable for acceptance testing/business users.
  • Setup/teardown can be ugly and complex if many processes/systems are enrolled in test. Many unit test frameworks do not support cooperative cancellation which would allow you to clean house.

Upvotes: 0

Ian Boyd
Ian Boyd

Reputation: 256771

The point of automated testing is to help you test.

You've taken the time to write a test. It is a good thing that it can be run again in the future (it helps to ensure you didn't break anything).

NUnit is a system to help you run those tests that you want to run. You're not a bad person for having "more thorough tests" mingled into with the "contrived tests".

The only downside is that some of those tests take too long; and a developer will be discouraged from running tests every time they run. That's when it's perfectly acceptable to categorize (i.e. selectively and temporarily) disable tests.

i have unit tests written that are specifically there to document a bug in a function. There's no immediate need to fix it since it's a one in a million chance that anyone will hit the bug (i've even been insulted for even testing the case). But at least the test is there - reminding everyone for all time that the function's not perfect:

void TestDateToString_KnownBug_FailsOnPseudo();

Sure i uncheck that test, so i don't see it fail all the time, but the point of unit testing is to make my life easier - not more painful.

Upvotes: 0

ckramer
ckramer

Reputation: 9443

I don't think there is anything wrong with writing your integration tests using the Unit Testing Framework you are already familiar with. Since these are integration tests there are a couple of things that differentiate them from your unit tests. One is the fact that they are dependent on external systems (even if you are spinning some of them up in your code), so if one of those is not available, your tests will fail. The other, as you have already pointed out, is the fact that integration tests take longer to run.

The way to deal with these issues is to configure your build scripts to not run the integration tests during your CI builds, but run them instead during a scheduled build (such as a nightly build). It is also important that the developers be able to run the integration tests manually on-demand, so that they can verify on their local machines that none of the tests have broken, and if they have, be able to verify that they have corrected the problem without manually triggering a system build.

Depending on the testing framework your using there are different ways of separating your unit tests from your integration tests so that you can configure your builds to execute one or both. One way is to move your integration tests into a separate project, and only execute tests in that project during your nightly build. The other is to use something like the Category attribute in NUnit to mark some tests as integration tests. You can then configure the test runner to exclude tests in this category for the builds that you do not want to execute your integration tests.

Upvotes: 3

Related Questions