Pragmatically allow User to pick which JUnit Test Classes will be run

Question

I am trying to pass a list of classes as a parameter. (Can I do this?) I am using JUnit and Selenium, I have JUnit test classes that are called by a JUnit test suite class, using @SuiteClasses() and that test suite class is called by a class containing a main(). My idea is to allow the user to pick JUnit classes from the main class which will be stored in some kind of list. The Test Suite that calls the JUnit test classes to be run will use that list and call those JUnit classes.

---

Original Code: the test suite class that calls the JUnit test classes that should be run (works) ⬇

import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;

@RunWith(Suite.class)
@SuiteClasses({ TestCase1.class, TestCase2.class})

public class AllTests {

}

I am trying to change it to something like ⬇

@SuiteClasses(runnerClass.classesToTest)

and in the runner class I would have something like this. I was thinking, I can pull names of classes from prop file maybe, and allow the user to pick which classes will be added to variable classesToTest ⬇

public class runnerClass {    
   public static Class<?>[] classesToTest = { testCase1.class, testCase2.class };
   public static void main(String[] args) {
      ...
   }
}

When I try to do something like this, I get this error ⬇

The value for annotation attribute Suite.SuiteClasses.value must be a class literal

JavaDoc for @SuiteClasses()

---

So question being, can I make this work? Am I creating my classesToTest variable incorrectly?

Answer 1

I could not find any solution in the JUnit framework, so I wrote a quick and dirty Test runner. It just calls all Methods annotated with @Test, even non-accessible ones (just in case).

It won't work with any IDE included UnitTest result displaying tools.

It is used like the following:

public static void main(String[] args) {
    Runner run = new Runner(TestCase.class, TestCase2.class);
    for(Exception e : run.runUnchecked()) {
        System.err.println(e.getCause());
    }
}

You can pass the Classes either as vararg or a normal array, both will work. The Runner will return a List of Exceptions of the tests. If a test fails, it throws an Exception, either the exception that caused the fail, or if an assertion failed, then a AssertionFailedError is thrown. You can easily print a one line description with e.getCause(), this will display a message like this: org.opentest4j.AssertionFailedError: expected: <1> but was: <2>

My example Code works with JUnit Jupiter tests, you can adapt it by changing which Test class is imported in the Runner class. This has to be the same, that is used for your TestCases.

Here is the Code

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import org.junit.jupiter.api.Test;

/**
 * A simple JUnit Test Case Runner, which can dynamically call TestCases by
 * their class.
 * 
 * @author Alexander Daum
 *
 */
public class Runner {
    private Class<?>[] testCases;

    public Runner(Class<?>... testCases) {
        this.testCases = testCases;
    }

    /**
     * Run all TestCases given in the constructor of this Runner.
     * 
     * @throws InvocationTargetException
     * @throws IllegalArgumentException
     * @throws IllegalAccessException
     * @throws InstantiationException
     */
    public List<Exception> run()
            throws IllegalAccessException, IllegalArgumentException, InvocationTargetException, InstantiationException {
        List<Exception> testErrors = new ArrayList<>();
        for (Class<?> testClazz : testCases) {
            Object testCase = testClazz.newInstance();
            Method[] methods = testClazz.getDeclaredMethods();
            methods = Arrays.stream(methods).filter(m -> m.isAnnotationPresent(Test.class)).toArray(Method[]::new);
            for (Method m : methods) {
                m.setAccessible(true);
                try {
                    m.invoke(testCase);
                } catch (InvocationTargetException e) {
                    testErrors.add(e);
                }
            }
        }
        return testErrors;
    }

    /**
     * The same as {@link Runner#run()}, but all exceptions are wrapped in
     * RuntimeException, so no try catch is neccessary, when Errorhandling is not
     * required
     */
    public List<Exception> runUnchecked() {
        try {
            return run();
        } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException
                | InstantiationException e) {
            throw new RuntimeException(e);
        }
    }
}