What layer of abstraction should be tested

Question

Let's say we have some functionality to write. It should validate csv file with some company data - one company per row. I'd like to write tests for this functionality. Today I had some conversation with my colleague and we can't decide which approach is better or maybe we are both wrong:

1. Writing tests at the level of some service which is entry point of validation. It has one method:

   ProcessingResult process(CsvFile csvFile)
   

   The test for IBAN number looks like this:

   // given
   valid csv row with invalid value in IBAN column
   // when
   service called
   // then
   result is validation failure with INVALID_IBAN error reason
   

2. Writing tests for specific validators. This approach assumes that there are few validators responsible for some groups of columns ex. CompanyNameValidator, AccountNumberValidator and so on.

First approach is good because it let us to refactor whole validation process without touching tests. I tried it and completely changed implementation running tests after every step and I was pretty confident I'm not breaking anything. This approach has cons as well. Let's say I have a test for company name and for IBAN. Consider implementation checking company name before IBAN. Let's say this name check is broken. Now test for IBAN is broken as well (the company name broke it). Is it ok or not?

Second approach won't have above problem as the company name is validated by different validator than IBAN. However it makes bigger refactoring impossible. If we wanted to change implementation and resign from multiple validators approach we would have to modify tests as well as we are testing them not the whole functionality.

Which approach is better or maybe there is another way to test this kind of code I'm not aware of?

Answer 1

The main subject I enjoy teaching my teams:

Do your best to avoid coupling your tests with implementation details because unit test, especially in a TDD context, is about refactoring, not testing.

Testing specific validators will lead to brittle tests because the capacity of refactoring will be severely altered.

Indeed, what if tomorrow you or your teammate have a better way of designing code that involve a different nature of validation?

What if tomorrow you let a third party library do the validation work?
What if tomorrow you want to combine/merge some validators?

My point is: Always unit test at the API level, like the first approach you mentioned.
This way, you will be able to refactor your code without breaking any test.

Your test should not be aware of the strategy you use to make the work done.
It should only be focused with the entry point: the API (generally a public method in Java/C# etc).

People complain about unit testing wasting their time; and the symptom is unique:
In their way, tests are coupled to the implementation details; so break too easily during the refactoring step.

Don't write tests files for every validation separately, unless your listening of your tests when something breaks becomes too difficult to figure out.
Resist to do it as long as you can otherwise you'll fall into the trap where you won't want to refactor since you won't "allow" your existing tests break (a kind of sad human behaviour).

Let's say I have a test for company name and for IBAN. Consider implementation checking company name before IBAN. Let's say this name check is broken. Now test for IBAN is broken as well (the company name broke it).

This point does clearly not depend upon a testing strategy.
Your algorithm should be adapted according to the related user story:
Are validation "fail-fast" ones (in that case, failure in checking company name interrupted the whole process) or not?
Just write another test where the context is a successful company name checking and a failed IBAN validation; but keep targeting the API.

If you want some additional information, I suggest you to watch that video; the best I've ever watched on the subject.

Answer 2

First approach will bring more value, as you noticed, because you can freely play with implementation.

Let's say I have a test for company name and for IBAN. Consider implementation checking company name before IBAN. Let's say this name check is broken. Now test for IBAN is broken as well (the company name broke it). Is it ok or not?

Write tests for every validation test case separately. Then you will have clean "test report" where name of test will tell you exact problem you have without reading/opening failed test message.

However because you reading files, suggest to abstract file reading, then your tests become little bid more faster and do not affect on the maintaining of validation logic.