Unit Testing Strategy, Ideal Code Coverage Baseline

Question

There's still not much information out there on the XCode7 and Swift 2.0 real-world experiences from a unit testing and code coverage perspective.

While there're plenty of tutorials and basic how-to guides available, I wonder what is the experience and typical coverage stats on different iOS teams that actually tried to achieve a reasonable coverage for their released iOS/Swift apps. I specifically wonder about this:

1) while code coverage percentage doesn't represent the overall quality of the code base, is this being used as an essential metric on your team? If not, what is the other measurable way to assess the quality of your code base?

2) For a bit more robust app, what is your current code coverage percentage? (just fyi, we have hard time getting over 50% for our current code base)

3) How do you test things like:

• App life-cycle, AppDelegate methods
• Any code related to push/local notifications, deep linking
• Defensive programming practices, various piece-of-mind (hardly reproducible) safe guards, exception handling etc.
• Animations, transitions, rendering of custom controls (CG) etc.
• Popups or Alerts that may include any additional logic

I understand some of the above is more of a subject for actual UI tests, but it makes me wonder:

• Is there a reasonable way to get the above tested from the UTs perspective? Should we be even trying to satisfy an arbitrary minimal code coverage percentage with UTs for the whole code base or should we define that percentage off a reasonably achievable coverage given the app's code base?
• Is it reasonable to make the code base more inflexible in order to achieve higher coverage? (I'm not talking about a medical app where life would be in stake here)
• are there any good practices on testing all the things mentioned above, other than with UI tests?

Looking forward to a fruitful discussion.

Answer 1

You do ask a very big and good question. Although your question includes:

I wonder what is the experience and typical coverage stats on different iOS teams ...

I think the issue is language/OS agnostic. Sure some languages and platform are more unit testable than others. So some are more expensive to unit test (as opposed to other forms of automated/coded testing). I think you are searching for a cost/benefit equation to maximize productivity. Ah the fun of software development processes.

To jump to the end to give you the quick sound grab answer:

You should unit test all code that you want to work and is appropriate to unit testing.

So now why the all and why the emphasis on unit testing ...

What is a unit test?

The language in the development community is corrupted, so please bear with me. Unit testing is just one type of automated testing. Others are Automated Acceptance Tests, Application tests, Integration Tests, and Components test. These all test different things. They have different purposes.

However, when I hear unit testing two things pop into mind:

1. What is a unit test?
2. As part of TDD (Test Driven Development)?

TDD is about writing tests before writing code. It is a very low level coding practice/process (XP - eXtreme Programming) as you write a test to write a statement and then another test. Very much a coding practice but not an application/requirements practice as it is about writing code that does what you intended, not what the product requirements are (oh gosh I feel the points being lost).

Writing code and then unit testing it is ... in my experience ... fun, short term team building, but not productive. Sure some defects are found, but not many. TDD leads to better "healthy" code.

My point here is that unit testing is:

1. A subset of automated/coded testing.
2. Is part of a coding process.
3. Is about code health (maintainability).
4. Does note prove that your application works (sound of falling points).

Why all?

If you're team delivers zero defect software (ZDFD is real and achievable .. but that a flat earth discussion) all the time without unit testing then this is nonsense and you would not be asking any questions here.

The only valid reason for a team to include unit testing as part of its coding process is to improve productivity. If all team members commit to team productivity then the only issue is identifying which code profits from unit testing. This is the context of the all.

The easiest way I think to illustrate this is to list types I do not unit test:

• Factories - They only instantiate types.
• Builders / writing (IoC) - Same as factories - No domain logic.
• Third party libraries - We call 3rd party libraries as documented. If you want to test these then use integration/component tests.
• Cyclomatic Complexity of one - Every method of of type has a CC of 1. That is, no conditions. Unit tests will tell you nothing useful, peer review is more useful.

The practical answer

My teams have expected 100% unit test coverage on all new code that should be unit tested. This is achieved by attributing code that does not meeting the unit testing criteria. All code must go through code review and the attributes must be specific to the why options listed above. -- Simple.

A long answer, and perhaps not easy to digest, nor what people want to hear. But, from long experience, I know it is the best answer that can lead to best profitability.

Post comment

My answer is aimed at the unit testing aspects of the question. As for defensive programming and other practices, TDD is a process that mitigates that by making it harder to do the wrong thing. But build system static code analysis tools may help you capture these before they get to peer review (they can fail a build on new issues). Look at others like SonarQube, Resharper, CppDepend, NDepend (yes language dependent).