Black Box Unit Testing

Question

In my last project, we had Unit Testing with almost 100% cc, and as a result we almost didn’t have any bugs.

However, since Unit Testing must be White Box (you have to mock inner functions to get the result you want, so your tests need to know about the inner structure of your code) any time we changed the implementation of a function, we had to change the tests as well.

Note that we didn't change the logic of the functions, just the implementation.

It was very time-consuming and it felt as if we are working the wrong way. Since we used all proper OOP guidelines (specifically Encapsulation), every time we changed the implementation we didn't have to change the rest of our code, but had to change the unit tests.

It felt as if we are serving the tests, instead of them serving us.

To prevent this, some of us argued that unit tests should be Black Box Testing.

That would be possible if we create one big mock of our entire Domain and create a stub for every function in every class in one place, and use it in every unit test.

Of course that if a specific test needs specific inner function to be called (Like making sure we write to the DB), we can override our stub.

So, every time we change the implementation of a function (like adding or replacing a call to a help function) we will only need to change our main big mock. Even if we do need to change some unit tests, it will still be much less than before.

Others argue that unit tests must be White Box, since not only do you want to make sure your app writes to the DB in a specific place, you want to make sure your app does not write to the DB anywhere else unless you specifically expect it to. While this is a valid point, I don't think it worth the time of writing White Box tests instead of Black Box tests.

So in conclusion, two questions:

1. What do you think about the concept of Black Box Unit Testing?

2. What do you think about the way we want to implement that concept? Do you have better ideas?

Answer 1

tl;dr version:

1. Black Box unit testing is exactly how unit testing should be done.
2. Black Box unit testing is exactly how unit testing should be done. Proper TDD practice does exactly this.

Full version.

There is absolutely no need in testing private methods of the objects. It'll have no impact on code coverage, also.

When you TDD a class, you write tests that check the behavior of that class. Behavior is expressed through the public methods of that class. You should never bother with how that methods are really implemented. Google people described that a lot better than I will ever be able to: http://googletesting.blogspot.ru/2013/08/testing-on-toilet-test-behavior-not.html

If you do the usual mistake and statically depend on other entity classes or worse, on classes from the different layer of application, it's inevitable that you will find yourself in a situation when you need to check a lot of things in your test and prepare a lot of stuff for it. For solving this the Dependency Injection principle and the Law of Demeter exist.

Answer 2

For detailed information about black, white and grey box and decision tables refer to the following article, which explains everything.

Testing Web-based applications: The state of the art and future trends (PDF)

Answer 3

"as a result we almost didn’t have any bugs" -- so keep it that way. Sole cause of frustration is necessity to maintain unit tests, which actually is not such a bad thing (alternative is much worse). Just make them more maintainable. "The art of Unit Testing" by Roy Osherove gave me a good start in this way. So 1) Not an option. (The idea itself contradicts principles of TDD, for instance) 2) You'll have much more maintenance troubles with such approach. Unit testing philosophy is to chop out SUT from other system and test it using stubs as input and mocks as output (signals?) simulating real life situations (or mb I just dont catch the "one big mock of our entire Domain" idea).

Answer 4

and as a result we almost didn’t have any bugs

If you were really able to achieve this, then I don't think you should change anything.

Black box testing might sound appealing on paper, but truth is you almost always need to know parts of inner workings of a tested class. The provide input, verify output in reality works only for simple cases. Most of the times your tests need to have at least some knowledge of tested method - how it interacts with external collaborators, what methods it calls, in what order and so forth.

Whole idea behind mocking and SOLID design is to avoid situation where dependency implementation change causes other class test changes/failures. On contrary, if you change implementation details of tested method, so should change implementation details of it tests. That's nothing too uncommon.

Overall, if you were really able to achieve almost no bugs, then I would stick to that approach.

Answer 5

You need different types of tests.

• Unit-tests which should be white-box testing, as you did

• Integration tests (or system tests) which test the ability to use the actual implementations of your system and its communication with external layers (external systems, database, etc.) which should be black-box styled, but each one for a specific feature (CRUD tests for example)

• Acceptance tests which should be completely black-box and are driven by functional requirements (as your users would phrase them). End-to-end as much as possible, and not knowing the internal of your chosen implementations. The textbook definition of black-box tests.

And remember code coverage is meaningless in most of the cases. You need a high lines coverage (or methods coverage, whatever your counting method is), but that's usually not sufficient. The concept you need to think about is functional coverage: making sure all your requirements and logical paths are covered.

Answer 6

I think you should continue writing unit tests - just make them less fragile.

Unit tests should be low level but should test the result and not how things done. When implementation change cause a lot of test change it means that instead of testing requirements you're actually testing implementation.

There are several rules of the thumb - such as "don't test private methods" and use mock objects.

Mocking/simulating the entire domain usually result in the opposite of what you're trying to accomplish - when the code behavior change you need to update the tests to make sure that your "simulated objects" behaves the same - it becomes really hard really fast as the complexity of the project increase.

I suggest that you continue writing unit tests - just learn how to make them more robust and less fragile.