CODEWITHSUNDEEP
javaseleniumtestng
mahan
mahan

Reputation: 15015

testng - skip multiple tests

I have a test a case that if fails, 100 more tests would fail. These 100 tests are in different test classes. It is cumbersome to use dependsOnMethod on all of these tests. Is there a more maintainable appreach to skip these 100 tests should the first test fail?

info:

Upvotes: 0

Views: 1134

Answers (2)

Krishnan Mahadevan
Krishnan Mahadevan

Reputation: 14746

Here's how you can do this:

  1. Create an implementation of IInvokedMethodListener and within its beforeInvocation check if an attribute (which is used to detect failures) has been set to indicate failures. If this attribute is set, then you mark your test method as skipped, set an exception to it and throw an exception from within beforeInvocation
  2. Within the afterInvocation() implementation, check if the test method that was executed failed. If yes, then set the failure attribute at the suite object level.

Just ensure that you don't run tests in parallel with this setup, because this logic will not work when tests run concurrently.

Below sample shows all of this in action.

Here's the failure detecting listener.

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;

public class EarlyFailureDetectingListener implements IInvokedMethodListener {
  private static final String SKIP_FURTHER_EXECUTION = "skipFurtherExecution";

  @Override
  public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
    if (testResult
        .getTestContext()
        .getSuite()
        .getAttributeNames()
        .contains(SKIP_FURTHER_EXECUTION)) {
      testResult.setStatus(ITestResult.SKIP);
      IllegalStateException exception = new IllegalStateException("Failure detected");
      testResult.setThrowable(exception);
      String msg =
          String.format(
              "Skipping execution of %s.%s()",
              method.getTestMethod().getTestClass().getRealClass().getName(),
              method.getTestMethod().getMethodName());
      System.err.println(msg);
      throw exception;
    }
  }

  @Override
  public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
    if (testResult.getStatus() == ITestResult.FAILURE) {
      testResult.getTestContext().getSuite().setAttribute(SKIP_FURTHER_EXECUTION, Boolean.TRUE);
    }
  }
}

Here's how the test classes look like:

import org.testng.Assert;
import org.testng.annotations.Test;

public class TestClassOne {

  @Test
  public void testMethod() {
    Assert.fail("Simulating a failure");
  }

  @Test
  public void anotherTestMethod() {
    TestClassTwo.printer();
  }
}

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.Test;

public class TestClassTwo {

  @Test
  public void testMethod() {
    printer();
  }

  @Test
  public void anotherTestMethod() {
    printer();
  }

  static void printer() {
    ITestResult result = Reporter.getCurrentTestResult();
    String msg =
        String.format(
            "Executing ====> %s.%s() on Thread [%d]",
            result.getTestClass().getRealClass().getName(),
            result.getMethod().getMethodName(),
            Thread.currentThread().getId());
    System.err.println(msg);
  }
}

import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.Test;

public class TestClassThree {

  @Test
  public void testMethod() {
    printer();
  }

  @Test
  public void anotherTestMethod() {
    printer();
  }

  static void printer() {
    ITestResult result = Reporter.getCurrentTestResult();
    String msg =
        String.format(
            "Executing ====> %s.%s() on Thread [%d]",
            result.getTestClass().getRealClass().getName(),
            result.getMethod().getMethodName(),
            Thread.currentThread().getId());
    System.err.println(msg);
  }
}

Here's the suite xml 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="55160447_suite" parallel="false" verbose="2">
  <listeners>
    <listener
      class-name="com.rationaleemotions.stackoverflow.qn55160447.EarlyFailureDetectingListener"/>
  </listeners>
  <test name="55160447_test_one">
    <classes>
      <class name="com.rationaleemotions.stackoverflow.qn55160447.TestClassOne">
      </class>
    </classes>
  </test>
  <test name="55160447_test_two">
    <classes>
      <class name="com.rationaleemotions.stackoverflow.qn55160447.TestClassTwo">
      </class>
    </classes>
  </test>
  <test name="55160447_test_three">
    <classes>
      <class name="com.rationaleemotions.stackoverflow.qn55160447.TestClassThree">
      </class>
    </classes>
  </test>
</suite>

Here's the execution output:

...
... TestNG 7.0.0-beta3 by Cédric Beust ([email protected])
...
Executing ====> com.rationaleemotions.stackoverflow.qn55160447.TestClassOne.anotherTestMethod() on Thread [1]

java.lang.AssertionError: Simulating a failure

    at org.testng.Assert.fail(Assert.java:97)
    at com.rationaleemotions.stackoverflow.qn55160447.TestClassOne.testMethod(TestClassOne.java:10)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:570)
    at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:170)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:790)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

PASSED: anotherTestMethod
FAILED: testMethod
java.lang.AssertionError: Simulating a failure
    at org.testng.Assert.fail(Assert.java:97)
    at com.rationaleemotions.stackoverflow.qn55160447.TestClassOne.testMethod(TestClassOne.java:10)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:570)
    at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:170)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:790)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)


Skipping execution of com.rationaleemotions.stackoverflow.qn55160447.TestClassTwo.anotherTestMethod()


===============================================Skipping execution of com.rationaleemotions.stackoverflow.qn55160447.TestClassTwo.testMethod()
    55160447_test_one
    Tests run: 2, Failures: 1, Skips: 0
===============================================

Test ignored.

Test ignored.
SKIPPED: anotherTestMethod
java.lang.IllegalStateException: Failure detected
    at com.rationaleemotions.stackoverflow.qn55160447.EarlyFailureDetectingListener.beforeInvocation(EarlyFailureDetectingListener.java:18)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker$InvokeBeforeInvocationWithoutContextStrategy.callMethod(InvokedMethodListenerInvoker.java:88)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker.invokeListener(InvokedMethodListenerInvoker.java:61)
    at org.testng.internal.BaseInvoker.runInvokedMethodListeners(BaseInvoker.java:55)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:547)
    at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:170)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:790)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

SKIPPED: testMethod
java.lang.IllegalStateException: Failure detected
    at com.rationaleemotions.stackoverflow.qn55160447.EarlyFailureDetectingListener.beforeInvocation(EarlyFailureDetectingListener.java:18)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker$InvokeBeforeInvocationWithoutContextStrategy.callMethod(InvokedMethodListenerInvoker.java:88)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker.invokeListener(InvokedMethodListenerInvoker.java:61)
    at org.testng.internal.BaseInvoker.runInvokedMethodListeners(BaseInvoker.java:55)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:547)
    at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:170)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:790)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)


===============================================
    55160447_test_two
    Tests run: 2, Failures: 0, Skips: 2
===============================================

Test ignored.

Skipping execution of com.rationaleemotions.stackoverflow.qn55160447.TestClassThree.anotherTestMethod()
Skipping execution of com.rationaleemotions.stackoverflow.qn55160447.TestClassThree.testMethod()

Test ignored.
SKIPPED: anotherTestMethod
java.lang.IllegalStateException: Failure detected
    at com.rationaleemotions.stackoverflow.qn55160447.EarlyFailureDetectingListener.beforeInvocation(EarlyFailureDetectingListener.java:18)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker$InvokeBeforeInvocationWithoutContextStrategy.callMethod(InvokedMethodListenerInvoker.java:88)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker.invokeListener(InvokedMethodListenerInvoker.java:61)
    at org.testng.internal.BaseInvoker.runInvokedMethodListeners(BaseInvoker.java:55)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:547)
    at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:170)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:790)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

SKIPPED: testMethod
java.lang.IllegalStateException: Failure detected
    at com.rationaleemotions.stackoverflow.qn55160447.EarlyFailureDetectingListener.beforeInvocation(EarlyFailureDetectingListener.java:18)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker$InvokeBeforeInvocationWithoutContextStrategy.callMethod(InvokedMethodListenerInvoker.java:88)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker.invokeListener(InvokedMethodListenerInvoker.java:61)
    at org.testng.internal.BaseInvoker.runInvokedMethodListeners(BaseInvoker.java:55)
    at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:547)
    at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:170)
    at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:790)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)


===============================================
    55160447_test_three
    Tests run: 2, Failures: 0, Skips: 2
===============================================

Test ignored.

===============================================
55160447_suite
Total tests run: 6, Passes: 1, Failures: 1, Skips: 4
===============================================


Process finished with exit code 0

Upvotes: 2

Thomas Hirsch
Thomas Hirsch

Reputation: 2346

While I would still suggest to use either dependsOnMethods or structure your tests differently to solve your problem, you could also skip these tests programatically.

For this purpose, set up a MethodInterceptor. In the TestContext you can call setAttribute to store a value indicating that your test failed, and then in the interceptor you can skip any method, by changing the list that it returns accordingly.

But like I said in the beginning, consider the additional complexity that this adds, compared to the rather simple dependsOnMethods approach. Remind yourself that tests also serve the purpose to document things as well.

Upvotes: -1

Related Questions