CODEWITHSUNDEEP
gradletestng
khris
khris

Reputation: 4999

How to re-run failed TestNG tests using Gradle

I know what TestNG generates testng-failed.xml with information about failed test. I'd like to use it in Gradle to created new task for running failed tests:

task secondTry(type: Test) {
onlyIf {
    file("build/reports/tests/test/testng-failed.xml").exists()
}
println file("build/reports/tests/test/testng-failed.xml").exists()

testClassesDir = sourceSets.test.output.classesDir
classpath = sourceSets.test.runtimeClasspath

useTestNG() {
    suites("build/reports/tests/test/testng-failed.xml")
}
}

Aftrer running task secondTry I got true for println file("build/reports/tests/test/testng-failed.xml").exists(), but failed tests are not run and task is skipped: secondTry SKIPPED

Does anyone experience in such cases, how to make tests run?

UPDATE: I've tried to modify task a bit, so just run testng-failed.xml:

task secondTry(type: Test) {
useTestNG() {
    useDefaultListeners = true
    reports.html.enabled = false
    options.suites("build/reports/tests/test/testng-failed.xml")
}
}

As result, build is successfully executed, including secondTry task, but failed tests still are not run.

Upvotes: 5

Views: 1982

Answers (3)

M. Justin
M. Justin

Reputation: 21399

The Test Retry Gradle plugin is designed to retry failed tests, and works with TestNG, JUnit 4 & 5, and Spock. It will rerun each failed test a certain number of times, with the option of failing the build if too many failures have occurred overall.

plugins {
    id 'org.gradle.test-retry' version '1.2.1'
}

test {
    retry {
        maxRetries = 3
        maxFailures = 20 // Optional attribute
    }
}

Upvotes: 0

Gaurav Singh
Gaurav Singh

Reputation: 43

The idiomatic way of doing this in TestNG is to implement the retry method in IRetryAnalyzer interface and then annotate the required test methods or class that you expect flaky tests to be in. TestNG will automatically run them again as part of your test suite. You can refer to the TestNG docs for this.

Here is a working example of this in Kotlin language, Here the failed test methods would be run again until they meet the specified criteria.

RerunTests.kt

import framework.core.retry.FailureRunner
import org.testng.Assert
import org.testng.annotations.Test


@Test(groups = ["rerun"], retryAnalyzer = FailureRunner::class)
class RerunTests {
    fun foobar() {
        Assert.assertFalse(true)
    }

    fun foo() {
        Assert.assertEquals(2, 2)
    }
}

And here is a sample implementation of a class that implements the required interface

FailureRunner.kt

import org.testng.IRetryAnalyzer
import org.testng.ITestResult
import org.testng.Reporter

class FailureRunner : IRetryAnalyzer {
    private var retryCounter = 0
    private var maxRetryCount = System.getProperty("maxFailureRetries").toInt()

    override fun retry(result: ITestResult?): Boolean {
        val testName = result?.testName
        return if (retryCounter < maxRetryCount) {
            Reporter.log("Current retry count <$retryCounter> is less than the max no of failure retries <$maxRetryCount>")
            Reporter.log("Retrying $testName")
            ++retryCounter
            true
        } else {
            Reporter.log("Retry count exceeded max limit for $testName.")
            false
        }
    }
}

Upvotes: 0

Prakash Saravanan
Prakash Saravanan

Reputation: 656

This can be achieved by a TestNG's feature called RetryAnalyzer. With few little tweaks, we can tell TestNG that how many times a test should get retried until it gets passed and what kind of exceptions/errors would trigger the retrying of the failed test methods.

Upvotes: 1

Related Questions