Reputation: 1615
I have a Java app with Maven and JUnit for tests, with failsafe and surefire plugins.
I have more than 2000 integration tests. To speed up the test running, I use failsafe jvmfork to run my tests in parallel. I have some heavy test class, and they typically running at end of my test execution and it is slows down my CI verify process. The filesafe runorder:balanced would be a good option for me, but I cant use it because the jvmfork. To rename the test classes or move to another package and run it alpahabetical is not an option. Any suggestion how can I run my slow test classes at the begining of the verify process?
Upvotes: 36
Views: 41419
Reputation: 24512
In JUnit 5 (from version 5.8.0 onwards) test classes can be ordered too.
src/test/resources/junit-platform.properties:
# ClassOrderer$OrderAnnotation sorts classes based on their @Order annotation
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$OrderAnnotation
Other Junit built-in class orderer implementations:
org.junit.jupiter.api.ClassOrderer$ClassName
org.junit.jupiter.api.ClassOrderer$DisplayName
org.junit.jupiter.api.ClassOrderer$Random
For other ways to set configuration parameters (beside junit-platform.properties file) see JUnit 5 user guide.
You can also provide your own orderer. It must implement ClassOrderer
interface:
package foo;
public class MyOrderer implements ClassOrderer {
@Override
public void orderClasses(ClassOrdererContext context) {
Collections.shuffle(context.getClassDescriptors());
}
}
junit.jupiter.testclass.order.default=foo.MyOrderer
Note that @Nested
test classes cannot be ordered by a ClassOrderer
.
Refer to JUnit 5 documentations and ClassOrderer API docs to learn more about this.
Upvotes: 59
Reputation: 24512
Here is another possible way which may be useful in some scenarios.
For example, if I want all my UI tests (which take a long time to execute) to run after my regular unit tests.
For this, a separate source set can be created and then making it run after regular tests.
build.gradle.kts:
sourceSets {
create("uiTest") {
// Adds files from the main source set to the compile classpath and runtime classpath of this new source set.
// sourceSets.main.output is a collection of all the directories containing compiled main classes and resources
compileClasspath += sourceSets.main.get().output
runtimeClasspath += sourceSets.main.get().output
}
}
// Makes the uiTest configurations extend from test configurations,
// which means that all the declared dependencies of the test code (and transitively the main as well)
// also become dependencies of this new configuration
val uiTestImplementation by configurations.getting {
extendsFrom(configurations.testImplementation.get())
}
val uiTestRuntimeOnly by configurations.getting {
extendsFrom(configurations.testRuntimeOnly.get())
}
val uiTest = task<Test>("uiTest") {
description = "Runs UI tests."
group = "verification"
testClassesDirs = sourceSets["uiTest"].output.classesDirs
classpath = sourceSets["uiTest"].runtimeClasspath
mustRunAfter("test") // The important part
testLogging {
events(TestLogEvent.PASSED)
}
}
tasks.check { dependsOn(uiTest) }
tasks.withType<Test> {
useJUnitPlatform() // Enables JUnit 5 (along with JUnit 4) tests
}
src
--- main
--- java
--- resources
--- test
--- java
--- resources
--- uiTest
--- java
--- resources
Create your UI tests in src/uiTest/java/...
directory like you would in src/test/java/...
Now, if you execute the check
Gradle task (./gradlew check
) (which is simply a task that executes other test tasks), the UI tests will be executed after the regular unit tests.
Sidenote:
./gradlew uiTest
task./gradlew test
taskUpvotes: 0
Reputation: 1207
There is a solution for that from version 5.8.0-M1 of junit.
Basically you need to create your own orderer. I did something like that.
Here is an annotation which you will use inside your test classes:
@Retention(RetentionPolicy.RUNTIME)
public @interface TestClassesOrder {
public int value() default Integer.MAX_VALUE;
}
Then you need to create class which will implement org.junit.jupiter.api.ClassOrderer
public class AnnotationTestsOrderer implements ClassOrderer {
@Override
public void orderClasses(ClassOrdererContext context) {
Collections.sort(context.getClassDescriptors(), new Comparator<ClassDescriptor>() {
@Override
public int compare(ClassDescriptor o1, ClassDescriptor o2) {
TestClassesOrder a1 = o1.getTestClass().getDeclaredAnnotation(TestClassesOrder.class);
TestClassesOrder a2 = o2.getTestClass().getDeclaredAnnotation(TestClassesOrder.class);
if (a1 == null) {
return 1;
}
if (a2 == null) {
return -1;
}
if (a1.value() < a2.value()) {
return -1;
}
if (a1.value() == a2.value()) {
return 0;
}
if (a1.value() > a2.value()) {
return 1;
}
return 0;
}
});
}
}
To get it working you need to tell junit which class you would use for ordering descriptors. So you need to create file "junit-platform.properties" it should be in resources folder. In that file you just need one line with your orderer class:
junit.jupiter.testclass.order.default=org.example.tests.AnnotationTestOrderer
Now you can use your orderer annotation like Order annotation but on class level:
@TestClassesOrder(1)
class Tests {...}
@TestClassesOrder(2)
class MainTests {...}
@TestClassesOrder(3)
class EndToEndTests {...}
I hope that this will help someone.
Upvotes: 6
Reputation: 27
You can use annotations in Junit 5 to set the test order you wish to use:
From Junit 5's user guide: https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
@TestMethodOrder(OrderAnnotation.class)
class OrderedTestsDemo {
@Test
@Order(1)
void nullValues() {
// perform assertions against null values
}
@Test
@Order(2)
void emptyValues() {
// perform assertions against empty values
}
@Test
@Order(3)
void validValues() {
// perform assertions against valid values
}
}
Upgrading to Junit5 can be done fairly easily and the documentation on the link in the beginning of the post contains all the information you might need.
Upvotes: -6
Reputation: 2565
Let me summarize everything before I will provide a recommendation.
At this moment we are sure we don't have a problem with getting a quick feedback. But we still want to run integration tests faster. I would recommend the following solutions:
I worked with many different projects (some of them had CI build running for 48 hours) and first 3 steps were enough (even for crazy cases). Step #4 is rarely needed having good tests. Step #5 is for very specific situations.
You see that my recommendation relates to the process and not to the tool, because the problem is in the process.
Quite often people ignore root cause and try to tune the tool (Maven in this case). They get cosmetic improvements but with high maintenance cost of created solution.
Upvotes: 4
Reputation: 4259
I gave the combination of answers I found a try:
The second answer is based on these classes
of this github
project, which is available under the BSD-2 license.
I defined a few test classes:
public class LongRunningTest {
@Test
public void test() {
System.out.println(Thread.currentThread().getName() + ":\tlong test - started");
long time = System.currentTimeMillis();
do {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
}
} while(System.currentTimeMillis() - time < 1000);
System.out.println(Thread.currentThread().getName() + ":\tlong test - done");
}
}
@Concurrent
public class FastRunningTest1 {
@Test
public void test1() {
try {
Thread.sleep(250);
} catch (InterruptedException e) {
}
System.out.println(Thread.currentThread().getName() + ":\tfrt1-test1 - done");
}
// +7 more repetions of the same method
}
Then I defined the test suites:
(FastRunningTest2 is a copy of the first class with adjusted output)
@SuiteClasses({LongRunningTest.class, LongRunningTest.class})
@RunWith(Suite.class)
public class SuiteOne {}
@SuiteClasses({FastRunningTest1.class, FastRunningTest2.class})
@RunWith(Suite.class)
public class SuiteTwo {}
@SuiteClasses({SuiteOne.class, SuiteTwo.class})
@RunWith(ConcurrentSuite.class)
public class TopLevelSuite {}
When I execute the TopLevelSuite
I get the following output:
TopLevelSuite-1-thread-1: long test - started
FastRunningTest1-1-thread-4: frt1-test4 - done
FastRunningTest1-1-thread-2: frt1-test2 - done
FastRunningTest1-1-thread-1: frt1-test1 - done
FastRunningTest1-1-thread-3: frt1-test3 - done
FastRunningTest1-1-thread-5: frt1-test5 - done
FastRunningTest1-1-thread-3: frt1-test6 - done
FastRunningTest1-1-thread-1: frt1-test8 - done
FastRunningTest1-1-thread-5: frt1-test7 - done
FastRunningTest2-2-thread-1: frt2-test1 - done
FastRunningTest2-2-thread-2: frt2-test2 - done
FastRunningTest2-2-thread-5: frt2-test5 - done
FastRunningTest2-2-thread-3: frt2-test3 - done
FastRunningTest2-2-thread-4: frt2-test4 - done
TopLevelSuite-1-thread-1: long test - done
TopLevelSuite-1-thread-1: long test - started
FastRunningTest2-2-thread-5: frt2-test8 - done
FastRunningTest2-2-thread-2: frt2-test6 - done
FastRunningTest2-2-thread-1: frt2-test7 - done
TopLevelSuite-1-thread-1: long test - done
Which basically shows that the LongRunningTest
is executed in parralel to the FastRunningTests
. The default value of threads used for parallel execution defined by the Concurrent
Annotation is 5
, which can be seen in the output of the parallel execution of the FastRunningTests
.
The downside is that theses Threads
are not shared between FastRunningTest1
and FastRunningTest2
.
This behavious shows that it is "somewhat" possible to do what you want to do (so whether that works with your current setup is a different question).
Also I am not sure whether this is actually worth the effort,
TestSuites
manually (or write something that autogenerates them)threads
for each class)As this basically shows that it is possible to define the execution order of classes and trigger their parallel execution, it should also be possibly to get the whole process to only use one ThreadPool
(but I am not sure what the implication of that would be).
As the whole concept is based on a ThreadPoolExecutor, using a PriorityBlockingQueue
which gives long running tasks a higher priority you would get closer to your ideal outcome of executing the long running tests first.
I experimented around a bit more and implemented my own custom suite runner and junit runner. The idea behind is to have your JUnitRunner submit the tests into a queue which is handeld by a single ThreadPoolExecutor
. Because I didn't implement a blocking operation in the RunnerScheduler#finish
method, I ended up with a solution where the tests from all classes were passed to the queue before the execution even started. (That might look different if there a more test classes and methods involved).
At least it proves the point that you can mess with junit at this level if you really want to.
The code of my poc is a bit messy and to lengthy to put it here, but if someone is interested I can push it into a github project.
Upvotes: 8
Reputation: 6705
In out project we had created a few marker interfaces ( example
public interface SlowTestsCategory {}
)
and put it into the @Category annotation of JUnit in the test class with slow tests.
@Category(SlowTestsCategory.class)
After that we created some special tasks for Gradle to run tests by category or a few categories by custom order:
task unitTest(type: Test) {
description = 'description.'
group = 'groupName'
useJUnit {
includeCategories 'package.SlowTestsCategory'
excludeCategories 'package.ExcludedCategory'
}
}
This solution is served by Gradle, but maybe it'll be helpful for you.
Upvotes: 6