How to ensure Unit Tests stay correct in this scenario?

Question

I don't really know how to concisely explain my situation, so I'll have to describe the scenario.

I have a process A that takes an input X and produces an output Y. I have a separate process B that takes the input Y (from process A) and produces an output Z.

Both processes A and B are complicated and so would benefit from having unit testing. It is also likely that processes A and B will change and thus the "middle" format Y will change.

If I simply have unit tests for B that have input Y, how do I ensure that they remain relevant and correct if process A changes? For example process A turns the input "foo" into "bar". Unit Tests for B have "bar" as their input and convert it into "overflow". If process A changes and now turns "foo" into "fish", then my unit tests for B will still pass but their worth is questionable since they are no longer testing expected input.

What is best practice to resolve this situation? Does this situation have a name?

Also (to add complexity), process B is Java but process A is Visual Basic.

I know that going "beyond" Unit Testing and performing Integration Testing would be a way to ensure that input X can become output Z but since we have a great deal of tests for process B, how can we ensure that these stay relevant? (and also it's not easy when one is Visual Basic and the other is Java).

Answer 1

Having different systems that talk to one another is a very common problem, and there's no one right solution. However in every instance of this problem there are two ways to address it, and your solution will be some balance of the two.

1. Test the interaction

   As you suggest, end-to-end integration tests are critical to the success of your application. They are more complicated and more prone to failure than unit tests, but they are all the more necessary for it. If you don't have proper integration tests (whatever "proper" means for your application) you cannot know that your application works at all - your unit tests will pass in a vacuum.

   How you do this will depend on how business-critical your application and its dependencies are. If your whole livelihood depends on credit card transaction being successfully and securely processed, you better have a test of some sort that verifies this; otherwise one day you'll wake up to angry pagers about millions of dollars in lost transactions, or worse a lawsuit about lost or exposed data. On the other hand, you want to select only the most important things to test in this way, since doing so is expensive and time consuming.

2. Accept volatile input

   The alternative is for your second program to accept as many inputs as possible, and properly handle them. You can unit test this. Design process B to be as fault-tolerant as possible, and write tests for as many failure cases as you can hope to handle. If it should handle foo and turn it into bar, what should it do about FoO? Or f o o? How should it respond if it receives no input at all? This sort of fault tolerance is a type of loose coupling.

   Generally speaking, failing fast and hard is desirable (i.e. accept a fixed set of inputs, and raise clear exceptions when you get something unexpected). But when you can't reliably define those inputs you have to handle all requests without failing. For instance if you're a user-visible webserver you can't hope to test all inputs, so instead you have to handle anything that might get thrown your way.

Depending on your needs, you'll want to rely more on option 1 or option 2. Without more context it's hard to say how much you should lean one way or the other, but hopefully this is enough to get you thinking about it.

Answer 2

If you're not ready to adopt integration tests, then a stop-gap measure would be to use data-driven unit tests. There are many ways to make this work; the one that struck me as I read the question was to:

1. use separate test input and test verification files for each process, and then
2. use the test verification file for process A as the test input file for process B

So when you update the verification file for process A, your tests for process B will immediately fail, reminding you to update your verification file for process B. And of course, when you update the test input file for process B, the tests for process A will immediately fail, reminding you to update your tests for process A.

Answer 3

I think you already know the answer: integration (end-to-end) testing. If setting that up is not feasible, the best you can do is test X/Y and Y/Z separately, and make sure that whenever one breaks (because of code changes), you update the other one to keep the test data relevant.

In my view, it's far more safer to invest in setting up a proper integration test suite, especially if that end-to-end flow is matching your user's expectations. In the end, they won't care about any of your nice unit tests, they just want to see your software doing a particular thing for their particular input. So writing end-to-end behavior tests is a lot more future-proof and also documents nicely what your software does.