How to decide test cases for unit tests?

Question

I'm just getting into unit testing, and have written some short tests to check if function called isPrime() works correctly.

I've got a test that checks that the function works, and have some test data in the form of some numbers and the expected return value.

How many should I test? How do I decide on which to test? What's the best-practices here?

One approach would be to generate 1000 primes, then loop through them all, another would be to just select 4 or 5 and test them. What's the correct thing to do?

Answer 1

Ask yourself: what EXACTLY do I want to test, and test the most important things. Test to make sure it basically does what you are expecting it to do in the expected cases.

Testing all those nulls and edge-cases - I - don't think is real, too time consuming and someone needs to maintain that later! And...your test code should be simple enough so that you do not need to test your test code!

If you want to check that your function correctly applied the algorithm and works in general - probably will be enough some primes.

If you want prove that the method for finding primes is CORRECT - 100000 primes will not be enough. But you don't want to test the latter (probably).

Only you know what you want to test!

PS I think using loops in unit tests is not always wrong but I would think twice before doing that. Test code should be VERY simple. What if something goes wrong and there is a bug in your test? However, you should try to avoid test code duplication as regular code duplication. Someone has to maintain test code.

Answer 2

"If it's worth building, it's worth testing"
"If it's not worth testing, why are you wasting your time working on it?"

I'm guessing you didn't subscribe to test first, where you write the test before you write the code?
Not to worry, I'm sure you are not alone.
As has been said above, test the edges, great place to start. Also must test bad cases, if you only test what you know works, you can be sure that a bad case will happen in production at the worst time.

Oh and "I've just found the last bug" - HA HA.

Answer 3

To be really sure, you're going to have to test them all. :-)

Seriously though, for this kind of function you're probably using an established and proven algorithm. The main thing you need to do is verify that your code correctly implements the algorithm.

The other thing is to make sure you understand the limits of your number representation, whatever that is. At the very least, this will put an upper limit on the size the number you can test. E.g., if you use a 32-bit unsigned int, you're never going to be able to test values larger than 4G. Possibly your limit will be lower than that, depending on the details of your implementation.

Just as an example of something that could go wrong with an implementation: A simple algorithm for testing primes is to try dividing the candidate by all known primes up to the square root of the candidate. The square root function will not necessarily give an exact result, so to be safe you should go a bit past that. How far past would depend on specifically how the square root function is implemented and how much it could be off.

Another note on testing: In addition to testing known primes to see if your function correctly identifies them as prime, also test known composite numbers to make sure you're not getting "false positives." To make sure you get that square root function thing right, pick some composite numbers that have a prime factor as close as possible to their square root.

Also, consider how you're going to "generate" your list of primes for testing. Can you trust that list to be correct? How were those numbers tested, and by whom?

You might consider coding two functions and testing them against each other. One could be a simple but slow algorithm that you can be more sure of having coded correctly, and the other a faster but more complex one that you really want to use in your app, but is more likely to have a coding mistake.

Answer 4

I've also been informed that every time a bug is found, you should write a test to verify that it is fixed. It seems reasonable to me, anyway.

Answer 5

You'd want to check edge cases. How big a prime number is your method supposed to be able to handle? This will depend on what representation (type) you used. If you're only interested in small (really relative term when used in number theory) primes, you're probably using int or long. Test a handful of the biggest primes you can in the representation you've chosen. Make sure you check some non-prime numbers too. (These are much easier to verify independently.)

Naturally, you'll also want to test a few small numbers (primes and non-primes) and a few in the middle of the range. A handful of each should be plenty. Also make sure you throw an exception (or return an error code, whichever is your preference) for numbers that are out of range of your valid inputs.

Answer 6

A few questions, the answers may inform your decision:

• How important is the correct functioning of this code?
• Is the implementation of this code likely to be changed in the future? (if so, test more to support the future change)
• Is the public contract of this code likely to change in the future? (if so, test less - to reduce the amount of throw-away test code)
• How is the code coverage, are all branches visited?
• Even if the code doesn't branch, are boundary considerations tested?
• Do the tests run quickly?

Edit: Hmm, so to advise in your specific scenario. Since you started writing unit tests yesterday, you might not have the experience to decide among all these factors. Let me help you:

• This code is probably not too important (no one dies, no one goes to war, no one is sued), so a smattering of tests will be fine.
• The implementation probably won't change (prime number techniques are well known), so we don't need tests to support this. If the implementation does change, it's probably due to an observed failing value. That can be added as a new test at the time of change.
• The public contract of this won't change.
• Get 100% code coverage on this. There's no reason to write code that a test doesn't visit in this circumstance. You should be able to do this with a small number of tests.
• If you care what the code does when zero is called, test that.
• The small number of tests should run quickly. This will allow them to be run frequently (both by developers and by automation).

I would test 1, 2, 3, 21, 23, a "large" prime (5 digits), a "large" non-prime and 0 if you care what this does with 0.

Answer 7

"Beware of bugs. I have proven the above algorithm correct, but have not tested it yet."

Some people don't understand the above quote (paraphrase?), but it makes perfect sense when you think about it. Tests will never prove an algorithm correct, they only help to indicate whether you've coded it right. Write tests for mistakes you expect might appear and for boundary conditions to achieve good code coverage. Don't just be picking values out of the blue to see if they work, because that might lead to lots of tests which all test exactly the same thing.

For your example, just hand-select a few primes and non-primes to test specific conditions in the implementation.

Answer 8

in general, test as many cases as you need to feel comfortable/confident

also in general, test the base/zero case, the maximum case, and at least one median/middle case

also test expected-exception cases, if applicable

if you're unsure of your prime algorithm, then by all means test it with the first 1000 primes or so, to gain confidence