CODEWITHSUNDEEP
javatestngtestcase
Marsh
Marsh

Reputation: 8125

Automatically retry all tests 3 times before failing them

We are using TestNG for our integration tests. We recently converted from jUnit, and we used to use a org.junit.rules.TestRule to automatically retry each test up to 3 times before counting it as failed. This eliminated a lot of false positives whenever a test case failed only occasionally.

In our conversion to TestNG, this retry rule was overlooked, and now we have a bunch of test cases "failing" that are really false positives.

I found a few articles on how to automatically re-run TestNG test cases:

https://jepombar.wordpress.com/2015/02/16/testng-adding-a-retryanalyzer-to-all-you-tests/

http://mylearnings.net/11.html

The gist of it is you can specify a retryAnalizer for each individual @Test-annotated test case. I set up my own analyzer and applied it to a test case, and that works. But applying a retry analyzer to every single test case manually is not a good solution, when we want every test case in the suite to do this. The article on jepombar.wordpress.com shows a way to apply it to all tests in a class, but for whatever reason it doesn't seem to work as written.

I made the following IAnnotationTransformer:

public class RetryListener implements IAnnotationTransformer {

    @Override
    public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
        IRetryAnalyzer retry = annotation.getRetryAnalyzer();
        if (retry == null) {
            annotation.setRetryAnalyzer(RetryRule.class); // my TestNG RetryAnalizer implementation
        }
    }

}

And I apply it to a class like this:

@Listeners(RetryListener.class)
public class FooTest extends SeleniumMockedTest {
    ...
}

This doesn't work; the code in RetryListener.transform() never executes, so RetryRule is never added to any of the test cases for the class.

How can I get this to work?

Or, better yet, my real question: How can I get all the test cases in our integration test suite to automatically try 3 times before failing counts as actually failing?

Upvotes: 1

Views: 1743

Answers (2)

alkakaushal
alkakaushal

Reputation: 154

Try this, Make 2 classes,

public class RetryAnalyzer implements IRetryAnalyzer {
int counter = 0;
@Override
public boolean retry(ITestResult result) {  
    RetryCountIfFailed annotation = result.getMethod().getConstructorOrMethod().getMethod()
            .getAnnotation(RetryCountIfFailed.class);
    result.getTestContext().getSkippedTests().removeResult(result.getMethod());
    if((annotation != null) && (counter < annotation.value()))
    {
        counter++;
        return true;
    }
    return false;
}

And write the other class to look like this

@Retention(RetentionPolicy.RUNTIME)
public @interface RetryCountIfFailed  {

    int value() default 0;
}

And now, pass the value of the count (retryCountGlobal) like below to every test you want to retry, 

 @Test
    @listeners.RetryCountIfFailed(retryCountGlobal)
    public void verifyRetryOnTestMethod(){
}

P.S : Remember that if retryCountGlobal=3, then the test will run 4 times, the first time if it fails, the test will be retried 3 more times.

Upvotes: 0

mfulton26
mfulton26

Reputation: 31234

I can not get it to work using @Listeners either but I can get it to work using the command line. e.g.:

java org.testng.TestNG -listener MyTransformer testng.xml

It not working using @Listeners may be a bug. You can report the issue here.

Upvotes: 1

Related Questions