One test watcher to report results of individual tests in JUnit Suite

Question

So I have a suite, something like this:

@RunWith(Suite.class)
@Suite.SuiteClasses({TestClass1.class, TestClass2.class, TestClass3.class})
public class TestSuite {

    static List<ExtentTest> extentTestList = new ArrayList<>();

    @ClassRule
    public static ExtentWatcher extentWatcher = new ExtentWatcher() {
        @Override
        protected void starting(Description description) {
            extentTestList.addAll(extentWatcher.getTests());
        }
        @Override
        protected void finished(Description description) {
            extentWatcher.flushReports(extentTestList);
        }
    };
}

The above code works, but the problem is that it results in my Watcher reporting the results of the Suite, not the individual tests. Also, if a test fails, the suite still reports as passed. My Watcher is something like this:

public class ExtentWatcher extends TestWatcher {

    // A little set up to give the report a date in the file name
    private DateFormat dateFormat = new SimpleDateFormat("dd-MM-yyyy");
    private Date date = new Date();
    private String fileDate = dateFormat.format(date);
    private String reportName = "./Test_Report_" + fileDate + ".html";
    public ExtentReports extent;
    ArrayList<ExtentTest> testList = new ArrayList<>();

    public ExtentWatcher() {
        extent = createReport();
    }

        // If test passed, watcher will record this with Extent Reports
        @Override
        protected void succeeded(Description description) {
            ExtentTest test = extent.startTest(description.getDisplayName());
            test.log(LogStatus.PASS, "Test Run Successful");
            testList.add(test);
        }

        // Likewise in case of failure
        @Override
        protected void failed(Throwable e, Description description) {
            ExtentTest test = extent.startTest(description.getDisplayName());
            test.log(LogStatus.FAIL, "Test Failure: " + e.toString());
            testList.add(test);
        }

    /**
     * These methods do the legwork - this file is based off of an example I found @ www.kieftsoft.nl/?p=81
     * Eventually we can add some screenshot logic here, but just a clean test report and version info will do
     */

    private ExtentReports createReport() {
        // Create the report - Extent just needs a little config
        ExtentReports extent = new ExtentReports(reportName, false);
        extent.config().reportName("Test Report: " + fileDate);
        return extent;
    }

    public void flushReports(List<ExtentTest> testList) {
        // This ends the test and then sends (flushes) everything to the html document
        for(ExtentTest test : testList) extent.endTest(test);
        extent.flush();
    }

    public List<ExtentTest> getTests() {
        return testList;
    }
}

This code works well annotated as @Rule for an individual test (with a report for each test individually, not desirable), but as per above, this isn't working on a Suite level and I'm really not sure how to make it work. I was thinking I could collect a list of all the tests, and in the suite, end the tests and flush them, which will allow ExtentReport to give me a report of all tests. However, I am unable to specifically get the individual test results - I will get one test back, with the displayName() = the Suite name.

How can I track the individual tests, then flush them when all have finished, and let the ExtentWatcher take care of the pass/fail on a test by test basis, instead of just once for the suite?

Answer 1

The above code works, but the problem is that it results in my Watcher reporting the results of the Suite, not the individual tests.

ClassRule specified in the TestSuite class are executed at the class level and they cannot be executed for each individual tests. Hence your watcher is not reporting for individual tests.

Since you are interested in listing the event for each individual test level, you may use the org.junit.runner.notification.RunListener (instead of Watcher). In order to avoid to specify the listener in each test class, you can implement your own Suite test runner which will register the custom RunListener automatically.

For example:

public class SuiteRunner extends Suite {
    public SuiteRunner(Class<?> klass) throws InitializationError {
        super(klass, new JUnit4Builder());
    }

    @Override
    public void run(RunNotifier notifier) {
        notifier.addFirstListener(new RunListener() {
            @Override
            public void testRunStarted(Description description) throws Exception {
                // add to watcher
            }

            @Override
            public void testFailure(Failure failure) throws Exception {
                // add to watcher
            }

        });
        super.run(notifier);
    }
}

And specify this runner in your Test Suite class

@RunWith(SuiteRunner.class)
@Suite.SuiteClasses({ MyTest.class , MyTest2.class })
public class TestSuite {
    @ClassRule
    public static ExtentWatcher watcher = new ExtentWatcher();
}

Hope it helps.

Answer 2

I use a RunListener implementation, which logs the results in a file. First, I have a test runner class that gets called for each test suite:

public Result run(String suite) {
    Class<?> suiteClass = null;
    try {
      suiteClass = Class.forName(suite);
    } catch (ClassNotFoundException ex) {
      return null;
    }
    JUnitCore core = new JUnitCore();
    this.testRun = new TestRun(testLogging, suite);
    core.addListener(new TestRunListener(testLogging, testRun));
    Result result = core.run(suiteClass);
    return(result);
}

TestRun class is a custom class that simply counts the number of tests that have: started, passed, failed, ignored.

TestRunListener implements org.junit.runner.notification.RunListener, and tracks information about test status, based on the callbacks (e.g. testFailure(), testFinished(), etc.).

@Override
public void testFailure(Failure failure) throws Exception {
  classLogger.log(Level.CONFIG, failure.getMessage());
  classLogger.log(Level.CONFIG, failure.getTestHeader());
  classLogger.log(Level.CONFIG, failure.getTrace());
  classLogger.log(Level.CONFIG, failure.getDescription().getDisplayName());
  if (failure.getException() != null) {
    classLogger.log(Level.CONFIG, failure.getException().getMessage());
  }
  super.testFailure(failure);

  Description description = failure.getDescription();
  Test test = processTestResult(TestStatus.FAIL, description);
  if (test == null) {
    classLogger.log(Level.SEVERE, "TEST IS NULL:" + description.getDisplayName());
    return;
  }

  classLogger.log(Level.CONFIG, failure.getMessage()+ " " + description.getDisplayName());

  test.setFailureMessage(description.getDisplayName());
  test.setFailureException(failure.getException());
  LogRecord record = new LogRecord(Level.CONFIG, failure.getMessage());
  record.setParameters(new Object[] { test, failure });
  testFailuresLogger.log(record);
}