CODEWITHSUNDEEP
javaunit-testingtestingjunitmockito
Usman Mutawakil
Usman Mutawakil

Reputation: 5259

Should unit tests only fail when your application is broken?

Question

Have you ever made changes to your application and everything runs just fine but when you go to run your unit tests all hell breaks lose? That is my specific problem. Would it be fair to say those are not good tests since they fail when my application is working fine? And by 'working' I mean it passes all acceptance tests and each feature works as intended.

Background

Most of the time after I've made changes to my application I can run through all the features with no problem, then when I run my unit tests I'll have failures. So there are instances in which I'm spending lots of time updating unit tests that failed for a perfectly working application. My gut tells me that if your tests fail, when the application passes, then they are useless tests and all they are doing is alerting the developer to implementation changes they have just made. Is this wrong? Can good tests fail when nothing else in the application in question has failed?

Example Scenario

You have unit tests that verify a certain filename is returned such as fileName.txt. Everywhere in your code that file is referred to using a constant IMPORTANT_FILENAME so changing the file name doesn't affect your application. You decide to change the name of the file from "fileName.txt" to "NewFileName.txt". Your application works fine as it should, but your unit test on verify(fileName,'fileName.txt') fails. This is a crude example but I hope you get the point. The point here is that when I find myself updating unit tests on a working application it seems to me its a red flag that is a bad test. I can't think of any test that fails in a passing application that is a good one. But before I use this as a rule of thumb and start deleting I wanted to get some feedback. Do you agree; unit tests shouldn't fail in a passing application?

Side-note: I don't think it matters but I use Java, Junit, and Mockito.

Upvotes: 2

Views: 2165

Answers (5)

Dawood ibn Kareem
Dawood ibn Kareem

Reputation: 79838

Unit tests alert you to those times when you've unwittingly broken some part of the application. Sometimes, they give you false alarms. But it's better to have occasional false alarms, than to let bugs into production.

So, yes, it's all right to have unit tests that fail sometimes because you've re-engineered the way your units interact with each other. It doesn't mean they're bad tests. It's better to have such tests than not to be alerted when you really do break your application.

Upvotes: 1

Dima
Dima

Reputation: 40500

In your example, yes, it is a bad test. If the API contract is to return the content of IMPORTANT_FILENAME, the test should be testing for that, and not a hardcoded string. In general, if a test is broken, the application does not have to necessarily fail. The purpose of the unit test is no so much to ensure the application functionality as it is to enforce the api contract. But even if it was the case, simple rules of logic will tell you that "application fails => tests are broken" is not equivalent to "tests are broken => application fails".

Upvotes: 3

wgitscht
wgitscht

Reputation: 2766

For me the main purposes of unit tests are

  • to give me certainty that a refactoring did not break anything
  • a functional change does not have any unwanted side-effects
  • safe me the time to test everything manually i.e. it may be easy to test a certain part of your code with a unit test but it's hard to reach from outside
  • often shows flaws in your design (if parts are hard to test automatically)
  • enforce programming paradigms

so if you change something vital to your application (i.e. change functionality) and the unit tests do not fail, you have bad unit tests. There may be for example a unit test to make sure that all fields are part of equals/hashcode. Or there may be unit tests to asure that certain classes are immutable. Your feature tests may still work, but you introduce unwanted code.

Another real important thing is the possibility to do TDD (Test Driven Development).

Upvotes: 2

dan b
dan b

Reputation: 1172

A good test can fail when nothing in the application has failed or could fail. Generally, unit tests should cover some aspect of an application that can occur. However, even if you are able to write test scripts that cover every scenario (a big assumption) it may be that a good unit test is not relevant to any such scenario. However, who knows whether someone may alter the application in which such a test would be relevant. Suppose you implement a Map class, but your application never uses the remove(Key) method. You have implemented a remove(key) unit tests however. Even though your application currently never uses the remove method, someone may come along at some point and use that method.

Upvotes: 0

dkatzel
dkatzel

Reputation: 31648

If you make changes to the application without changing any tests, I sure hope at least one test fails! The tests that verify the piece of code should break in some way. The rest of the tests should still all pass since they should be using mocks.

So you should have to make at least some minor changes to tests when something has changed.

However, if you are spending too much time having to fix your broken tests, you may have fragile tests. Without seeing the problem code, it's hard to say what to change, but you should apply as many Software Engineering principles and patterns to your test code as your production code. Try not to have copy and pasted code blocks. Reuse code via inheritance, helper methods or classes. That way if you need to make a change that affects many tests, you only have to make the change in one place.

Upvotes: 0

Related Questions