Would you call this (from the book "Code Complete") a unit test, an integration test or something else?

Question

In Steve McConnell's "Code Complete", he describes testing an encryption program:

I set up a test-data generator that fully exercised the encryption and decryption parts of the program. It generated files of random characters in random sizes. [...] For each random case, it generated two copies of the random file, encrypted one copy, reinitialized itself, decrypted the copy and then compared each byte in the decrypted copy to the encrypted copy. If any bytes were different, the generator printed all the information I needed to reproduce the error.

This test exhaustively tests the encryption "unit" and also performs some testing of other units - for example, those for file access and memory management.

I do not see this as a standard unit test - that would involve testing the encryption unit in isolation.

However, I also do not believe this is a standard integration test, since it concentrates so heavily on the encryption unit while using the others only in passing.

How would you classify tests similar to this one?

Answer 1

It is definitely not a unit test, since it is

• not testing the smallest possible unit, which would be a single method of the core encryption unit
• uses a lot of other functions as file access and memory management, instead of mocking them
• is too slow for a unit test.

Since the test is not going through the GUI (as mentioned in the comment below), it is not a system or acceptance test, unless the tests closely resemble customer requirements and drive the system under test just as if controlled through the GUI (for instance going under the hood by capturing the view's behavior from a MVC system and then removing the view by replaying the behavior).

So in most cases, I'd call it an integration test: the integration of the first 3 elements of {encryption unit, file access unit, memory management unit, GUI} are tested.

For well designed tests, the integration tests should focus on the interaction of the three units. The detailed functionality of each unit on its own should have been tested already in unit tests. Coming back to the quote you gave from Steve McConnell's "Code Complete", I would rather do the randomized encryption test by mocking the file access unit, and then on the integration test not have to test the core functionality of the encryption unit, but only test a couple of files (for situations like empty file; trivially encryptable file, e.g. only containing 'a'; hardly encryptable, e.g. already encrypted file; very large file; ...).

In short: Steve McConnell's test is a mixture of unit and integration test, I guess that's what caused you asking this question...

Answer 2

I'd call it a functional test or a system test (although only a partial one -- there needs to be another test that fixed, known test vectors produce the correct known output).

In principle, a unit test should execute no code from the project whatsoever, other than the unit under test. And possibly some other units that are independently tested without reference to the unit under test, and that are basically there to re-use code. This test doesn't take that approach.

Pedantically it could be considered a unit test of the encrypt executable, though, if the way it works is to run the encrypt program with a command line. I don't think that's the case here, your quote talks about "the encryption and decryption parts of the program", but it's a similar situation. Anyway, unless (maybe) you're RMS re-implementing Unix from scratch in the small hours of the morning, you don't consider an executable to be a "unit", you break things down much smaller than that.

Even aside from whether this test is constructed like a unit test, it may be that the encrypt "unit" is not truly a unit at all, but a collection of parts that themselves could be tested in isolation. By adding more dependency injection or otherwise redesigning the code, you could turn a less testable system into a more testable system and hence turn some of what used to be unit tests into integration tests. We don't know how testable this code really is. So perhaps you could describe it as an integration test of the encrypt module. It doesn't look like it came about as the result of a systematic attempt at integration testing, though, so it might be a bit misleading to call it that.

I think people often describe tests as "unit tests" even though really they aren't, and it's something most people can live with. Especially if you're using a test framework with "unit" in its name, it's easy/lazy to just describe that as "the unit tests", when in fact you're running several different kinds of tests in a batch.

Personally I don't think that the taxonomy of testing is hugely important, except that you shouldn't kid yourself that your tests are better than they really are, by thinking you've "unit tested everything" when really you haven't. I expect that if you're communicating with a large test team, the terminology becomes more important because everyone needs to know what they're talking about. But the largest test team I've ever worked with is only about 4 or 5 people.