CODEWITHSUNDEEP
javacucumberbddcucumber-jvm
Konstantin Dobroliubov
Konstantin Dobroliubov

Reputation: 669

Cucumber - How to mark expected fails as known issues?

I successfully use Cucumber to process my Java-based tests.

Sometimes these tests encounter regression issues, and it takes time to fix found issues (depending on issue priority, it can be weeks or even months). So, I'm looking for a way to mark some cucumber tests as known issues. Don't want these tests fail the entire set of tests, just want to mark them, for example, as pending with yellow color in report instead.

I know that I can specify a @tag for failed tests and exclude them from execution list, but that's not what I want to do as I still need these tests to be run continuously. Once the issue fixed, appropriate test should be green again without any additional tag manipulation.

Some other frameworks provide such functionality (run the test but ignore its result in case of fails). Is it possible to do the same trick somehow using Cucumber?

The final solution I use now - to mark known issues with specific tag, exclude these tests from regular round and to run them separately. But that's not the best solution I believe.

Any ideas appreciated. Thanks in advance.

Upvotes: 5

Views: 2646

Answers (2)

lauda
lauda

Reputation: 4173

You shouldn't.

My opinion is that if you have tests that fail then you should add a bug/task ticket for these scenarios and to add them in a build status page with the related tag. Another thing you could do is to add the ticket number as a tag and removed after the ticked is fixed.

If you have scenarios that fail due a bug, then the report should show that, if the scenario is not fully implemented then is better not to run it at all.

One thing you could do is to add a specific tag/name for those scenarios and to try in a before scenario method to get the tags and check for added specific tag/name and throw a pending exception.

What i suggest is to keep those scenarios running if there is a bug and to document that in the status page.

I think the client would understand better if those scenarios are red because they are failing rather than some yellow color with "grey" status of what is happening.

If you need the status of the run to trigger some CI job then maybe is better to change the condition there.

As i see it, the thing you need to give it a thought should be: what is the difference between yellow or red, pending or fail for you or the client?, you would wand to keep a clear difference and to keep track of the real status.

You should address these issues in an email, discuss them with the project team and with the QA team, and after a final decision is taken you should get a feedback from the customer also.

Upvotes: 1

Thomas Sundberg
Thomas Sundberg

Reputation: 4323

I would consider throwing a pending exception in the step that causes the known failure. This would allow the step to be executed and not be forgotten.

I would also consider rewriting the failing steps in such a way that when the failure occurs, it is caught and a pending exception is thrown instead of the actual failure. This would mean that when the issue is fixed and the reason for throwing the pending exception is gone, you have a passing suite.

Another thing I would work hard for is not to allow a problem to be old. Problems are like kids, when they grow up the they get harder and harder to fix. Fixing a problem while it is young, perhaps a few minutes, is usually easy. Fixing problems that are months old are harder.

Upvotes: 1

Related Questions