CODEWITHSUNDEEP
javamavenseleniumjenkinstestng
Tina Pan
Tina Pan

Reputation: 41

How to automatically re-run failed TestNG tests on Jenkins

I'm working on the last stages of building a Selenium automation suite for an enterprise environment. It's configured with TestNG, running on Jenkins. The suite consists mostly of UI tests. As typical of this setup the tests are flaky, and re-working the tests to be less dependent on UI automation is not feasible within our timeline.

Given that I'm already working on code optimization, I'd like to set something up that automatically re-runs failed tests to minimise the need for manual investigation of failures. I've investigated a couple of options, but none quite work for our setup:

Suggestions for a solution that would work with our setup would be appreciated.

Upvotes: 4

Views: 6075

Answers (3)

Scott Babcock
Scott Babcock

Reputation: 597

NOTE: This response replaces a previous response that got deleted by a moderator. See the next response for details of how to define and connect a TestNG retry analyzer. I'd prefer for my response to follow that one, but it shows up here instead.

In addition to the TestNG suite file method shown below, you can specify the retry analyzer is via a service loader configuration file. http://testng.org/doc/documentation-main.html#listeners-service-loader

This ServiceLoader mechanism has the advantage of working as expected regardless of the how the project is defined or how it gets run.

Upvotes: 0

Nael Marwan
Nael Marwan

Reputation: 1038

You can add Retry Listener:

Retry Class:

import org.testng.IRetryAnalyzer;

import org.testng.ITestResult;


public class Retry implements IRetryAnalyzer {

    private int retryCount = 0;

    private int maxRetryCount = 1;


    public String getResultStatusName(int status) {

        String resultName = null;

        if (status == 1)

            resultName = "SUCCESS";

        if (status == 2)

            resultName = "FAILURE";

        if (status == 3)

            resultName = "SKIP";

        return resultName;

    }


    /*

     * Below method returns 'true' if the test method has to be retried else

     * 'false' and it takes the 'Result' as parameter of the test method that

     * just ran

     * 

     * @see org.testng.IRetryAnalyzer#retry(org.testng.ITestResult)

     */

    @Override

    public boolean retry(ITestResult result) {

        if (retryCount < maxRetryCount) {

            System.out.println("Retrying test " + result.getName() + " with status " + getResultStatusName(result.getStatus()) + " for the " + (retryCount + 1) + " time(s).");

            retryCount++;

            return true;

        }

        return false;

    }

And add this listener:

import java.lang.reflect.Constructor;

import java.lang.reflect.Method;


import org.testng.IAnnotationTransformer;

import org.testng.IRetryAnalyzer;

import org.testng.annotations.ITestAnnotation;


public class RetryListener implements IAnnotationTransformer {


    @Override

    public void transform(ITestAnnotation testannotation, Class testClass, Constructor testConstructor, Method testMethod) {

        IRetryAnalyzer retry = testannotation.getRetryAnalyzer();


        if (retry == null) {

            testannotation.setRetryAnalyzer(Retry.class);

        }

    }


}

You have 2 options to call above listener:

From your xml testng runner as bellow:

<listeners>

        <listener class-name="package path....com.listeners.RetryListener" />

        <listener class-name="org.uncommons.reportng.HTMLReporter" />

        <listener class-name="org.uncommons.reportng.JUnitXMLReporter" />

</listeners>

From the test class:

@Listeners({ package path....com.listeners.RetryListener })

You need to add above declaration before the class declaration.  

Upvotes: 1

Sumit
Sumit

Reputation: 2426

For the failed testNG tests, the failure report is stored as testngfialed.xml. Add that to your XML Suite, something like this should work:

<suiteXmlFiles>
  <suiteXmlFile>src/test/<your path>.../testng.xml</suiteXmlFile>
  <suiteXmlFile>target/surefire-reports/testngfailed.xml</suiteXmlFile>
</suiteXmlFiles>

This might hep: Run testngfailed.xml via Jenkins

Upvotes: 0

Related Questions