CODEWITHSUNDEEP
unit-testingtestingtddintegration-testing
johnrl
johnrl

Reputation: 583

Test: stub vs real implementation

I have been wondering about the general use of stubs for unit tests vs using real (production) implementations, and specifically whether we don't run into a rather nasty problem when using stubs as illustrated here:

Suppose we have this (pseudo) code:

public class A {
  public int getInt() {
    if (..) {
      return 2;
    }
    else {
      throw new AException();
    }
  }
}

public class B {
  public void doSomething() {
    A a = new A();
    try {
      a.getInt();
    }
    catch(AException e) {
      throw new BException(e);
    }
  }
}


public class UnitTestB {
  @Test
  public void throwsBExceptionWhenFailsToReadInt() {
     // Stub A to throw AException() when getInt is called
     // verify that we get a BException on doSomething()
  }
}

Now suppose we at some point later when we have written hundreds of tests more, realize that A shouldn't really throw AException but instead AOtherException. We correct this:

public class A {
  public int getInt() {
    if (..) {
      return 2;
    }
    else {
      throw new AOtherException();
    }
  }
}

We have now changed the implementation of A to throw AOtherException and we then run all our tests. They pass. What's not so good is that the unit test for B passes but is wrong. If we put together A and B in production at this stage, B will propagate AOtherException because its implementation thinks A throws AException.

If we instead had used the real implementation of A for our throwsBExceptionWhenFailsToReadInt test, then it would have failed after the change of A because B wouldn't throw the BException anymore.

It's just a frightening thought that if we had thousand of tests structured like the above example, and we changed one tiny thing, then all the unit tests would still run even though the behavior of many of the units would be wrong! I may be missing something, and I'm hoping some of you clever folks could enlighten me as to what it is.

Upvotes: 2

Views: 860

Answers (5)

alkalinity
alkalinity

Reputation: 2030

Ancient thread, I know, but I thought I'd add that JUnit has a really handy feature for exception handling. Instead of doing try/catch in your test, tell JUnit that you expect a certain exception to be thrown by the class.

@Test(expected=AOtherException)
public void ensureCorrectExceptionForA {
    A a = new A();
    a.getInt();
}

Extending this to your class B you can omit some of the try/catch and let the framework detect the correct usage of exceptions.

Upvotes: 0

Gishu
Gishu

Reputation: 136633

The specific example you have mentioned is a tricky one.. the compiler cannot catch it or notify you. In this case, you'd have to be diligent to find all usages and update the corresponding tests.

That said, this type of issue should be a fraction of the tests - you cannot wave away the benefits just for this corner case.

See also: TDD how to handle a change in a mocked object - there was a similar discussion on the testdrivendevelopment forums (linked in the above question). To quote Steve Freeman (of GOOS fame and a proponent of the interaction-based tests)

All of this is true. In practice, combined with a judicious combination of higher level tests, I haven't seen this to be a big problem. There's usually something bigger to deal with first.

Upvotes: 0

guillaume31
guillaume31

Reputation: 14072

It's normal that the test you wrote using stubs doesn't fail since it is intended to verify that object B communicates well with A and can handle the response from getInt() assuming that getInt() throws an AException. It is not intended to check if getInt() really throws an AException at any point.

You can call that kind of test you wrote a "collaboration test".

Now what you need to be complete is the counterpart test that checks if getInt() will ever throw an AException (or a AOtherException, for that matter) in the first place. It's a "contract test".

J B Rainsberger has a great presentation on the contract and collaboration tests technique.

With that technique here's how you'd typically go, solving the whole "false green test" problem :

  1. Identify that getInt() now needs to throw a AOtherException rather than an AException

  2. Write a contract test verifying that getInt() does throw a AOtherException under given circumstances

  3. Write the corresponding production code to make the test pass

  4. Realize you need collaboration tests for that contract test : for each collaborator using getInt(), can it handle the AOtherException we're going to throw ?

  5. Implement those collaboration tests (let's say you don't notice there's already a collaboration test checking for AException at that point yet).

  6. Write production code that matches the tests and realize that B already expects an AException when calling getInt() but not a AOtherException.

  7. Refer to the existing collaboration test containing the stubbed A throwing an AException and realize it's obsolete and you need to delete it.

This is if you start using that technique just now, but assuming you adopted it from the start, there wouldn't be any real problem since what you'd naturally do is change the contract test of getInt() to make it expect AOtherException, and change the corresponding collaboration tests just after that (the golden rule is that a contract test always goes with a collaboration test so with time it becomes a no-brainer).

If we instead had used the real implementation of A for our throwsBExceptionWhenFailsToReadInt test, then it would have failed after the change of A because B wouldn't throw the BException anymore.

Sure, but this would have been a whole other kind of test -an integration test, actually. An integration test verifies both sides of the coin : does object B handle response R from object A correctly, and does object A ever respond that way in the first place ? It's only normal for a test like this to fail when the implementation of A used in the test starts to respond R' instead of R.

Upvotes: 0

Greg Philpott
Greg Philpott

Reputation: 54

If you change the interface of class A then your stub code will not build (I assume you use the same header file for production and stub versions) and you will know about it.

But in this case you are changing the behaviour of your class because the exception type is not really part of the interface. Whenever you change the behaviour of your class you really have to find all the stub versions and check if you need to change their behaviour as well.

The only solution I can think of for this particular example is to use a #define in the header file to define the exception type. This could get messy if you need to pass parameters to the exception's contructor.

Another technique I have used (again not applicable to this particular example) is to derive your production and stub classes from a virtual base class. This separates the interface from the implementation, but you still have to look at both implementations if you change the behaviour of the class.

Upvotes: 0

Tim S
Tim S

Reputation: 5101

When you say

We have now changed the implementation of A to throw AOtherException and we then run all our tests. They pass.

I think that's incorrect. You obviously haven't implemented your unit test, but Class B will not catch AException and thus not throw BException because AException is now AOtherException. Maybe I'm missing something, but wouldn't your unit test fail in asserting that BException is thrown at that point? You will need to update your class code to appropriately handle the exception type of AOtherException.

Upvotes: 2

Related Questions