Unit testing for signal processing?

Question

I came across similar problems like this one: Test driven development for signal processing libraries

The fact behind the problem is that the output of signal processing is hard to be fully and qualitatively defined.

So the inject input > run program > verify output approach does not apply well to signal processing.

Does meeting the performance requirement in the specifications mean that there's no bugs? Of course not. Then if it meets the requirement, why bother? Because bugs will bite, someday, they will.

In the end, the only feasible solution is to compare the output with a known-good equivalent, usually a matlab version or some widely used libraries.

Matlab has a good collection of libraries, and it has boundary checking, memory management, and it's double precision, so comparing with matlab code exposes pointer-out-of-bound, stack-overflow, and evaluates integer precision sufficiency, but it does not answer the question like "what if the matlab equivalent did it wrong, too?"

I can only say to myself: try to write the matlab equivalent simple, so it's close to "so simple that there's obviously no bug"

In my team, I have programmers at a variety of skill levels, and I need at least some kind of measure to control/enforce the quality of the code.

It's been more than two years since the last post. I with there are some new development in this area.

Please share with me, as practitioners, your ideas and opinions.

Answer 1

There are two problems here worth discussing:

1. Design decomposition.
2. Practical implementation and support

They're closely related, but different aspects to addressing TDD in DSP applications.

If in principle it sounds reasonable to decompose a filter design "to multiply blocks", it is actually pretty impractical to achieve the desired testing. It is equivalent to asking to decompose a string search to character comparisons, or a multiply into a shift-test-and-add block. It might be possible, but whether the tests actually then offer the opportunity to test meaningful units or expose likely algorithm or coding errors is doubtful.

An interesting practical test may want to examine the transient and steady state filter response, which might be the minimal vector operation in a filter implementation that exercises a meaningful monolithic block of code, and allows more subtle behavior to be tested - filter noise figures for instance. The practical challenge is providing a vector data input and output in the context of a unit test.

By extension looking at the result of an adaptive of PR problem requiring large data or state information become a similar but heterogeneous problem.

My ad hoc answer has been to build a test loop that drives the DSP and test against files defining test cases. Inherently off-line and limited to non-native hw, currently. Are there alternatives that folk use to move the test closer to the target.

Answer 2

GNU Radio is a stream-oriented signal processing framework which comes with a large library of signal processing algorithms, most of the time encapsulated in blocks.

Of course, we do a lot of unit testing, and we frequently find bugs, regressions or corner cases based on those, so this is crucial to the GNU Radio development process. I'm not absolutely convinced I'd call our approach TDD, but I don't think this is the point here.

The point is that of course signal processing can be unit-tested. Why shouldn't it? You specify inputs, and there's expected output, especially if you're encapsulating your steps in signal processing blocks. Now, whilst this is not as easy as your first-lesson unit test of the kind assert(square(2)==4), as a developer of a signal processing algorithm you have to be so well-aware of what you're supposed to do, and how you can mathematically verify it, that usually test cases can be written.

Now, I can understand your point of "my test case gets so complex, it might have bugs of its own", but that is just an indication that you have not yet structured your problem to subproblems that are small enough. As you've noticed, I like to cite GNU Radio, so let's do it again: A good test-driven development process will try to yield as many reusable, well-tested modules as possible. Well-tested implies that their behaviour is so well-specified that you can test it. GNU Radio does this in the form of simple blocks like "multiply", or more complex one like "schmidl&cox OFDM sync". You start of with verifying your simpler ones, and then verify the complete system, in that case "do the info bits that I feed into my OFDM modulator come out of my OFDM modulator". This works in signal processing as well as in any other genre -- it just demands a high level of understanding, which often is something that people with a pure computer science background have to acquire in the process of writing signal processing software (signal processing people, on the other hand, often struggle with finding good solutions to problems well-known and readily-solved to CS experts); I always hope this fact encourages people to write better tests (and read more books, too ;) ).