CODEWITHSUNDEEP
ctesting
tpg2114
tpg2114

Reputation: 15100

How to design software tests for subtle errors

This is a super simple example, in C here, to illustrate a subtle error that I don't know how to expose as a bug through a test.

Consider:

#include <stdio.h>

int main() 
{
  int a;
  int b;
  int input;

  printf("Enter 1 or 2: ");
  scanf("%d", &input);

  switch(input) {
  case 1:
    a = 10;
    /* ERROR HERE, I FORGOT A BREAK! */
  case 2:
    b = 20;
    break;
  default:
    printf("You didn't listen!\n");
    return 1;
    break;
  }

  if(input == 1) {
    b = 30;
    printf("%d, %d\n", a, b);
  } else {
    printf("%d\n",b);
  }

  return 0;
}

As noted in the code, a break is missing so when 1 is entered, it falls through to case 2. The output for 1 though doesn't reflect this as it overwrites b later. So all tests that we can design, say by entering a number from the set {1, 2, 10} all result in the correct output.

In reality, the assignments inside the switch could be very expensive and so this bug could be quite costly. But, assuming it was written this way from day one, there's no benchmark to see that the cost is higher than expected.

So what can be done to flush out these kinds of errors? Is there a way to design test cases to expose it in production software?

EDIT So I guess I wasn't entirely clear -- I wrote it in C to illustrate the type of problem encountered, but in reality it's not specific to C. The point I'm trying to make is that the code goes into sections we never intended it to go into (in this case because of a forgotten break to illustrate the point). My actual case is a Fortran code with 700,000 lines and it is going into branches we never intended it to go into because of poor if/switch design that is legal from a language point of view but potentially very expensive.

Is it possible to design a test or look at some data from some tool that will tell us it's going into branches it shouldn't be? I caught a mistake by printing "I shouldn't be here!" inside all the cases and saw that it was printed, there's gotta be a better way than randomly seeing it and putting print statements.

Upvotes: 2

Views: 382

Answers (5)

Klas Lindb&#228;ck
Klas Lindb&#228;ck

Reputation: 33273

Why is this a bug I ask? Using any kind of black-box testing the code works. So, if the only requirement is that the code works, then there is no bug.

But the code is flawed. That missing break makes the code more difficult to understand and harder to maintain.

Coding conventions are rules on how the code should look. Adherence to coding conventions can not be done by testing the compiled program, they must be done on the source code.

Testing coding conventions is done through code inspection (automatic or manual).

Edit:

If you are worried about performance, then use instrumentation tools to find the "hot spots". You will find that most of the execution time is probably spent in just a few modules. Review those modules and every call to them. You will find that you only need to review 10-30 000 lines of code. Since the review scope is limited, the review should take 1-3 weeks.

Bottom line: Code review is vastly superior to testing when it comes to finding subtle bugs.

Upvotes: 1

Jo So
Jo So

Reputation: 26511

For your specific example, it is in no way a mistake / bug by definition. The possibility to fallthrough is wanted in the language. If you want to forbid certain dangerous language features, then linters are the way to go.

To avoid completely unforeseen mistakes, there is the one rule: Always code defensively and make use of asserts whereever you can put them.

Upvotes: 1

vpit3833
vpit3833

Reputation: 7951

If input==1 and you see b = 30 on the output, you know something is wrong. Also, remember in the else clause, you should write something to b before reading from it. In the case of default:, (say, input==100) you might end up reading from a location without properly setting it.

Besides, code reviews, if you can afford them should help greatly in finding things like these.

Upvotes: 0

xaxxon
xaxxon

Reputation: 19771

The correct state for case 1 is that b won't be set.

Check to see if b is set.

You may need to break your code down into smaller segments to test this if you're setting b later, but that's just good modularity.

It seems like you're asking "how do I test untestable code?". The answer is that it takes skill and planning to write testable code, it can't just be an afterthought.

There's tons of stuff on the web to help you write testable code:

https://www.google.com/search?q=writing+testable+code

Upvotes: 1

Israel Unterman
Israel Unterman

Reputation: 13510

You can define coding convention for switch statements, so that each branch will impose a special state. Like a variable getting assigned the very case`s value. For example:

switch (v) {
case 1: 
    vcheck = 1;
    ...
    break;
case 2:
    vcheck = 2;
    ...
    break;
}

And test vcheck in your test case.

Other than that you can use tools that perform static code analysis of validation of MISRA rules - and engage them into your build process. They will induce some piece of mind... :-)

Finally, (my favorite) you can write a script that checks for such cases and warns agains them.

Upvotes: 1

Related Questions