IRetryAnalyzer produces incorrect results for test cases defined as SoftAssert

Question

I have a implementation of Retry Logic for failed test cases using IRetryAnalyzer and there are 2 types of asserts - Assert and SoftAssert defined in my test cases. IRetryAnayzer works fine for normal Assert however does not work as expected in case of SoftAssert. Below are the scenario details about the issue faced:

• If a test case which is defined as SoftAssert fails in first attempt and passes in next attempt, it continues to retry till the max retry attempt even though the test case is passing. In this case, if the next test case which is defined as normal assert (non-softassert) passes, it will also get marked as fail even though it is passing and will be retried for max retry attempt defined.
• If all test cases are defined as normal assert, it works as expected i.e if it fails in first attempt and passes in next attempt, it moves on and does not get stuck in retry loop.
• If a test case defined as SoftAssert is passing in first attempt, it does not retry and moves on to next test cases i.e it works as expected.

I need to keep few test cases as softAssert as I have to continue with the test run. Ex:

@Test(retryAnalyzer = RetryAnalyzer.class, groups = { "group1" }, priority=1)
    public void TestSection1(){
        Class1.verifyingAppLaunch(); //Defined as Assert
        Class1.Test1(); //Defined as softAssert
        Class1.Test2(); //Defined as softAssert
        Class1.Test3(); //Defined as softAssert
        Class1.Test4(); //Defined as softAssert
        Class1.Test5(); //Defined as softAssert
        softAssert.assertAll();
    }

Below is a sample of IRetryAnalyer and ListenerAdapter implementation. ListenerAdapter is implemented to remove duplicate test case execution which were marked as skipped as part of retry implementation. In the below sample code, if samplecondition1 fails in first attempt, it will retry for max retry count defined even if it passes in second attempt and will also mark the samplecondition2 as fail even if it is passing:

MyTestListenerAdapter.class

import java.util.Iterator;
import org.testng.ITestContext;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;
import org.testng.TestListenerAdapter;

public class MyTestListenerAdapter extends TestListenerAdapter {

    @Override
    public void onFinish(ITestContext context) {
        Iterator<ITestResult> skippedTestCases = context.getSkippedTests().getAllResults().iterator();
        while (skippedTestCases.hasNext()) {
            ITestResult skippedTestCase = skippedTestCases.next();
            ITestNGMethod method = skippedTestCase.getMethod();
            if (context.getSkippedTests().getResults(method).size() > 0) {
                System.out.println("Removing:" + skippedTestCase.getTestClass().toString());
                skippedTestCases.remove();
            }
        }
    }
}

TestRetryAnalyzer.class

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class TestRetryAnalyzer implements IRetryAnalyzer {
    int counter = 1;
    int retryMaxLimit = 3;

    public boolean retry(ITestResult result) {
        if (counter < retryMaxLimit) {
            counter++;
            return true;
        }
        return false;
    }
}

TestRetryTestCases.class

import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(MyTestListenerAdapter.class)
public class TestRetryTestCases {
SoftAssert softAssert = new SoftAssert();

    @Test(retryAnalyzer = TestRetryAnalyzer.class)
    public void firstTestMethod() {
        System.out.println("First test method");
        if (samplecondition1 == true)
            softAssert.assertTrue(true);
        else
            softAssert.assertTrue(false);
softAssert.assertAll();
    }

    @Test(retryAnalyzer = TestRetryAnalyzer.class)
    public void secondTestMethod() {
        System.out.println("Second test method");
        if (samplecondition2 == true)
            Assert.assertTrue(true);
        else
            Assert.assertTrue(false);
    }
}

Answer 1

I am not sure what version of TestNG are you working with, but I cant seem to reproduce this issue using TestNG 7.0.0 (the latest released version as of today).

You have an additional problem in your code. You are having the SoftAssert as a global variable. SoftAssert by its very implementation, remembers all the failures. So for every retry, it continues to persist all the failures from the first attempt till now. That means that a @Test method which involves a RetryAnalyser and which uses SoftAssert wherein there's a possibility of something failing, would cause that test method to never pass.

When using SoftAssert, you should always declare and use the SoftAssert object within the @Test method, so that it gets instantiated (and thus reset for every retry).

Here's the same sample that you shared (i tweaked it just a little bit) which demonstrates that this works fine in 7.0.0

As you can see from the output, its only the firstTestMethod (which has SoftAssert is being retried) and secondTestMethod (which has a hard assert and hasn't failed) is not being retried.

Test class (I have only altered this, everything else is borrowed from your original post)

import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
import org.testng.asserts.SoftAssert;

@Listeners(MyTestListenerAdapter.class)
public class TestRetryTestCases {
  int softAssertCounter = 0;
  int hardAssertCounter = 0;

  @Test(retryAnalyzer = TestRetryAnalyzer.class)
  public void firstTestMethod() {
    SoftAssert softAssert = new SoftAssert();
    System.out.println("First test method");
    if (softAssertCounter++ > 2) {
      softAssert.assertTrue(true);
    } else {
      softAssert.assertTrue(false);
    }
    softAssert.assertAll();
  }

  @Test(retryAnalyzer = TestRetryAnalyzer.class)
  public void secondTestMethod() {
    System.out.println("Second test method");
    if (hardAssertCounter++ < 2) {
      Assert.assertTrue(true);
    } else {
      Assert.assertTrue(false);
    }
  }
}

** Retry analyser with some additional logging **

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class TestRetryAnalyzer implements IRetryAnalyzer {
  int counter = 1;
  int retryMaxLimit = 3;

  public boolean retry(ITestResult result) {
    if (counter < retryMaxLimit) {
      counter++;
      System.err.println("Retrying the test method " + result.getMethod().getMethodName());
      return true;
    }
    return false;
  }
}

Console output

First test method
Retrying the test method firstTestMethod
Retrying the test method firstTestMethod

Test ignored.

First test method


Test ignored.

First test method


java.lang.AssertionError: The following asserts failed:
    did not expect to find [true] but found [false]

    at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:47)
    at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:31)
    at com.rationaleemotions.stackoverflow.qn58072880.TestRetryTestCases.firstTestMethod(TestRetryTestCases.java:22)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:584)
    at org.testng.internal.TestInvoker.retryFailed(TestInvoker.java:204)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:58)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:804)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:145)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at org.testng.TestRunner.privateRun(TestRunner.java:770)
    at org.testng.TestRunner.run(TestRunner.java:591)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:402)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:396)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1180)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1102)
    at org.testng.TestNG.runSuites(TestNG.java:1032)
    at org.testng.TestNG.run(TestNG.java:1000)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)


Second test method

Removing:[TestClass name=class com.rationaleemotions.stackoverflow.qn58072880.TestRetryTestCases]
Removing:[TestClass name=class com.rationaleemotions.stackoverflow.qn58072880.TestRetryTestCases]

===============================================
Default Suite
Total tests run: 2, Passes: 1, Failures: 1, Skips: 0
===============================================


Process finished with exit code 0