JUnit's @Ignore

Question

I wonder if it is a good practice to use JUnit's @Ignore. And how people are using it?

I came up with the following use case: Let's say I am developing a class and writing a JUnit test for it, which doesn't pass, because I'm not quite done with the class. Is it a good practice to mark it with @Ignore?

I'm a little concerned that we might miss the ignored test case later on or that people start using it to "force" tests to pass CI.

Answer 1

Thats pretty much fine, I suppose.

The docs says,

Test runners will report the number of ignored tests, along with the number of tests that ran and the number of tests that failed.

Hence, it means even if you forget to remove that afterwards, you should have been notified about that.

The example given in the docs, is completely resembling your case.

@Ignore("not ready yet")

Answer 2

I think that it a perfectly good way to use it.

Your CI server should be green (or blue in Hudson's case) all the time. Whenever it isn't your first priority is to fix it.

Now, if CI broke because of a bug in the test code (perhaps the test code is naughty and non-deterministic) then you should just ignore the test "@Ignore(This test code is borken, raised defect #123)" and raise the bug in your defect tracker.

You won't ship broken code because whenever you ship, you review all defects and decide if any of them are show stoppers right? A broken test that isn't running, will be considered along with the code / feature it was testing. You ship if and only if you're happy that the code it was testing is not also broken. If it ain't tested, consider it broken.

I'm hoping the junit xml report formatter, used when running tests from ant, will one day include the ignored count (and the reasons) along with pass, fail, and error. Maybe then, CI vendors will include the ignored test counts (if not I may have to write a Hudson plugin...)

Answer 3

@Ignore can be used on tests written for some third party services. I recently found myself in need to check if external service is sending back the data I expect. It was very convenient to do that in test (mocks). Since I knew that my input data won't work forever and I may need to run similar test later I added @Ignore instead of deleting test.

Answer 4

I think it's ok to use @Ignore's when the test relies on external entities that can't be mocked. We still need the test to make sure things are working but we dont't want to deploy a test which relies on an external dependency.

For example, if you write a custom SpreadsheetWriter/Reader for Google docs it makes sense to use a real google doc to test it with. However you wouldn't want your build to fail if google docs are down for whatever reason. After confirming that your unit test passes locally I would add an @Ignore before pushing it to production.

In general,I think @Ignore should be avoided as much as possible because in most cases external classes/systems can be mocked.

Answer 5

I think this goes against the core fundamentals of test driven development. Granted if you're not doing TDD then it probably doesn't matter so much.

I would say though that if you're using @ignore then you're not LEAN and not doing TDD.

If you're doing TDD then you shouldn't ever have failing tests in any commit that you make to your repo.

Am I totally crazy or does this make sense?

Answer 6

I routinely use @Ignore for tests which fail because of a known bug. Once the bug is acknowledged and logged in the bug databases, the test failure serves no purpose, since the bug is already known.

Still it makes sense to keep the test code, because it will be useful again once the bug is fixed. So I mark it to be ignored, with a comment indicating the related bug, and ideally note in the bug report that the test should be reactivated to test the fix.

Answer 7

IMHO, Ignore that should not be used lightly... due to the broken windows effect.

I rarely find myself using this attribute/annotation in xUnit. The only few times I've used them is as a TODO when writing TestCase#1, I see another test case(s) that I missed but which should also be included. Just so that I dont forget it, I write a small test case with a descriptive name and mark it with Ignore. Proceed to complete TestCase#1. But this is all intra-check-in. I never check in tests marked with Ignore.

However usually I just use a piece of paper - test list to jot down the new test case - which is much simpler. This also caters to the scenario where I'm partially done... completed 5 of 10 tests. Instead of checkin in 5 Ignored tests, I'd keep the test-list around and check in 5 passing tests. The assumption is that you'll complete the rest in the next few check-ins before jumping to something new.

Other 'special cases' I can think of is..
When you're waiting for a component from another team/person/vendor (whose interface has been published-agreed to), without which the tests can't run. In this case, you can write the tests and mark it with Ignore("Waiting on X to deliver component Y")

Answer 8

I think using @ignore is OK as long as there is -

1. A good reason why the method cannot be tested in some form and it is documented as such in the code. This should be a special case and warrant a discussion or code review to see if there is any way to test it.
2. The test is not yet built - this should ideally happen only for legacy code. This should be also subject to code review and tasks should be put to add tests.

That's the rules at least in my mind ;-)

Answer 9

Well, if you're not done with the class it's good the test fails. Marking it as @Ignore would mean you will ship a code with unfinished class. And right, maybe you're not using that class yet in any code that gets executed, but someday some other developer might see that class and use it. Then he fails even it should work.

I wouldn't use @Ignore in that case for sure.