writing dynamic unit tests (flexunit) is confusing. how do i make this more modular?

Question

i've been a long time lurker here getting help and now i can't find an answer for my current problem.

back info

i am writing some unit tests (yay!). i have 40 objects that implement an interface. the one function in that interface accepts two parameters, one Rectangle and one array of Rectangle:

public function foobar(foo:Rectangle, bar:Array/*Rectangle*/):void;

i want to write tests for each of these 40 objects. to make sure i test all possibilities, i need to run tests where there are variations of foo and variations of bar (in length and content). so x number of foo and 1 to x number of Rectangle in foo.

each object that implements the interface is running an algorithm that will do some calculations on each of the Rectangle in bar and change their properties. each algorithm produces drastically different results.

if i choose to have 10 possible foo objects, and 10 possible objects for the bar array, i would end up writing thousands! of tests. i do not want to have to hand write thousands of tests.

questions

would it be too backwards for me to write an algorithm that takes the possible objects, and runts tests on all possible configurations producing the results, and then i go back and hand check that all the results are correct? is that just the wrong way to do unit tests?

is it wrong to run an algorithm that produces results, THEN hand check the output?

my other thought would be that i feed an algorithm the possible objects, and it spits outs some xml or json that is formatted for the test harness, then i go through each test, fill in the missing assertion values, then feed them through?

my other plan would be to write an algorithm that accepts a list of foo Rectangle and a list of possible Rectangle to be used in bar, and have that algorithm produce JSON in a format that works with my test harness (it includes the assertions). since the algorithm producing the JSON won't know the assertions, i would hand write those in before sending it through the test harness.

is this a common practice?

thanks for any feedback :)

Answer 1

It is not easy to come up with a good answer, not knowing any details about the kind of calculations you are doing in your implementations, but I admire your willingness to get into unit testing, so I'll try my best anyway, and I hope the answer doesn't get too long. ;)

0. Great expectations?

To be honest, there probably isn't an answer that exactly matches your question - many ways to do a thing will do the job correctly, and the only fundamental rule that applies to unit tests is that they should reliably help you to prove your system is stable. If they don't do that, you shouldn't bother writing them. But if that could be done by creating an Excel sheet with a million lines of different input and output value combinations, and feeding that in CSV format to a for-loop in a unit test ...

OK, maybe there is a better way. But in the end it all depends on how thoroughly you want to do this, and how much you are willing to deviate from what you've already done to make your tests better.

1. Get prepared for some smart-ass remarks

From what I read between the lines, you haven't spend a lot of time thinking about testability, because you've written your code before writing the tests. Sadly, that's really not the best way of doing unit tests: Each line you add to production code should always be covered by a failing unit test before you even write it. Only that way can you always be sure your system works - and it's always testable! Sounds tiresome? It isn't, once you get used to it.

I won't bother you too much with fundamentals, but if you're really serious about unit tests, just let me recommend you start applying TDD to all future projects: To get started, perhaps watch the TDD episodes on cleancoders.com - Uncle Bob does a way better job of explaining these things than I do, and he's fun to watch (though his demos are in Java, but that shouldn't be much of a problem - the fundamental principles of TDD apply to all languages).

Meanwhile, I'll still make a few smart-ass remarks based on your question. Sue me ;)

2. Smart-ass remark #1: How to test?

Make sure you remember that the goal of your tests is to prove that the code you are testing works correctly, and not to repeat proving it for every possible combination of arguments. You should always keep the number of assertions to the lowest necessary to prove that your code is correct.

This, then, will answer your first question: You should have only one test to prove correctness for each algorithm you are testing. Different combinations of input and output values can be used for assertions within that test.

The only reason to add more tests per algorithm is when you test for failure, i.e. what happens if you pass null as an argument, or anything that violates constraints. Each time you would throw an error in case of failure, it should be tested in a separate test.

What's a bit more complicated, though, is choosing at which level of abstraction you start to write your tests. It's usually not necessary to write a test for each method in a class, especially not for private ones. This is another reason to apply TDD - it makes you think of what you're trying to do from the outside in, i.e. you test what a part of your system is supposed to do, instead of testing each implementation detail. When you test before you code, it's easy to add a test here and there when you notice that your program has grown and things get more complicated; it's always much harder to do this "after the fact".

3. Smart-ass remark #2: What to test?

The goal of your program design should be to make your units as decoupled from other parts of the system, as possible. This means that applying a combination of things to another combination of things in one unit is probably not good design. You should be able to test only the code implemented in the unit you are testing, separate from all other things. This means

• making sure each method you are testing does only one thing(!) and

• all the other things needed in that method must either be passed in as arguments, or provided to the class as field variables - let me make that clear: No creation of objects within your method, unless they are temp variables or return values! External dependencies, then, you should replace with test doubles when testing the method.

4. Feeble attempt to apply this to your problem

Why am I telling you all this? It seems to me like your approach is more like an integration test: There is a black box to test, and a gzillion things could come out of it. This is OK for some part, but you should still try to make that black box as small as possible.

Now, since I don't know anything about the actual math you are doing, I will start making a couple of assumptions from here on out. I'm sorry if these don't fit, but I'll be glad to add or change info, if you provide some code samples.

Obvious first guess: You're repeatedly applying the same calculation to all members of the bar array, based on the coordinate values of the foo Rectangle. This would mean that you're actually doing two things in your method: a) iterating over the bar array and b) applying a formula:

public function foobar ( foo:Rectangle, bar:Array ) : void {
    for each ( var rect:Rectangle in bar) {
        // things done to rect based on foo
    }
}

If this is the case, you can easily improve your architecture. The first step would be to isolate the formula:

public function foobar ( foo:Rectangle, bar:Array ) : void {
    for each ( var rect:Rectangle in bar) {
        applyFooValuesToRect( foo, rect);
    }
}

public function applyFooValuesToRect ( foo : Rectangle, rect : Rectangle ) : void {
    // things done to rect based on foo
} 

Now you'll see that what you should really be testing is the applyFooValuesToRect method - which suddenly makes writing your test a whole lot easier.

I can also imagine that there might be variations on the iteration: One implementation applies foo to all of bar, one matches against some criteria and applies only to positive matches, maybe one does a chain of calculations on foo based on each of the bar values, one may use two formulae instead of one, etc. If any of this applies to your project, you can greatly improve your API, and reduce the complexity, by using the Strategy pattern. For each of the 40 variations, make the actual formula a separate class that implements a common Formula interface:

public interface Formula {
    function applyFooToBar (foo:Rectangle, bar:Rectangle) : Rectangle;
}

public class FormulaOneImpl implements Formula {

    public function applyFooToBar (foo:Rectangle, bar:Rectangle) : Rectangle {
        // do things to bar
        return bar;
    }
}

public class FormulaTwoImpl implements Formula ... // etc.

You can now test each of the formulae separately, and apply assertions to the value returned.

Your original class will take a field variable of type Formula:

public class MyGreatImpl implements OriginalInterface {
    public var formula:Formula;
    //..
    public function foobar (foo:Rectangle, bar:Array):void {
        for each (var rect:Rectangle in bar) formula.applyFooToBar (foo, rect);
    }
}

You can then pass in all sorts of formulae - as long as they implement the interface. As a result, you can now use the interface to create mock objects for testing all the other parts of the algorithm: All a Formula mock object has to do is verify that applyFooToBar is called, and return a predetermined value you set for each assertion. That way you can make sure that you're really not testing the formula when you're testing the iteration of your array, for example.

In fact, you can try to apply this to other things, too: A CriteriaMatcher also makes for a nice Strategy, to start with...

When you break down your code like this, you might see that you have more than one implementation that rely on the same formula, but have different variations of the iteration loop, etc. It will probably even allow you to reduce the number of implementations of your initial interface - because the variations are now encapsulated into the different strategy classes.

Boy, this is a long text. I'll stop ranting now. I hope I could help you a bit with this - just comment or edit your question again, if I was too far off with my assumptions. Perhaps we can narrow down on a possible solution some more.