CODEWITHSUNDEEP
javaintellij-ideajunitintegration-testing
Daniel Kaplan
Daniel Kaplan

Reputation: 67440

How do I define an order to run junit tests in Intellij?

I have a flaky junit test that only fails if I run all my tests. I think that one test is causing another test to fail, I want to prove it before I try to fix it.

If I run all tests, it runs the "test pollution test" then it runs the "test that fails after test pollution". It also runs a lot of irrelevant, slow tests in between. But if I use a pattern to only run these two, it runs "test that fails after test pollution" then "test pollution test". As a result, both pass.

How do I only run "test pollution test" and "test that fails after test pollution", in that order?

Note: These tests are in separate classes.

Upvotes: 19

Views: 31219

Answers (5)

Michael Morrison
Michael Morrison

Reputation: 61

Just adding a handy tip
You can define the run order in intellij for a run configuration by Ctrl-clicking each test class in the order you want to run them then right clicking the class and selecting run tests (Ctrl+Shift+F10). If you inspect the run configuration it will be using "Pattern" as the resource type with a value that has the classes in the order you selected them:

com.example.Test1Class||com.example.Test2Class||com.example.Test3Class

This should help you confirm the theory about test pollution

Upvotes: 6

The Prototype
The Prototype

Reputation: 274

In JUnit 5 (from version 5.8.0 onwards) test classes can be ordered too.

src/test/resources/junit-platform.properties:

# ClassOrderer$OrderAnnotation sorts classes based on their @Order annotation
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$OrderAnnotation

Other Junit built-in class orderer implementations:

org.junit.jupiter.api.ClassOrderer$ClassName
org.junit.jupiter.api.ClassOrderer$DisplayName
org.junit.jupiter.api.ClassOrderer$Random

You can also provide your own orderer. It must implement ClassOrderer interface:

package foo;
public class MyOrderer implements ClassOrderer {
    @Override
    public void orderClasses(ClassOrdererContext context) {
        Collections.shuffle(context.getClassDescriptors());
    }
}

junit.jupiter.testclass.order.default=foo.MyOrderer

Note that @Nested test classes cannot be ordered by a ClassOrderer.

Refer to JUnit 5 documentations and ClassOrderer API docs to learn more about this.

source: JUnit test class order

I recommend paying attention: https://stackoverflow.com/a/68269239/10663941

method order: https://www.baeldung.com/junit-5-test-order (because I got here in search of an answer)

Upvotes: 3

Ali Dehghani
Ali Dehghani

Reputation: 48153

According to JUnit's wiki:

By design, JUnit does not specify the execution order of test method invocations. Until now, the methods were simply invoked in the order returned by the reflection API. However, using the JVM order is unwise since the Java platform does not specify any particular order, and in fact JDK 7 returns a more or less random order. Of course, well-written test code would not assume any order, but some do, and a predictable failure is better than a random failure on certain platforms.

From version 4.11, JUnit will by default use a deterministic, but not predictable, order (MethodSorters.DEFAULT). To change the test execution order simply annotate your test class using @FixMethodOrder and specify one of the available MethodSorters:

@FixMethodOrder(MethodSorters.JVM): Leaves the test methods in the order returned by the JVM. This order may vary from run to run.

@FixMethodOrder(MethodSorters.NAME_ASCENDING): Sorts the test methods by method name, in lexicographic order.

You could use MethodSorters.NAME_ASCENDING and change your method names to match with your specific order. I know you're using this just for debugging sake but it's a Test Smell to rely on your test methods execution order and JUnit does not provide more finer grain control over test methods execution order

Upvotes: 34

Abish R
Abish R

Reputation: 1567

As said by Ali Dehghani, You can order the test method execution by

@FixMethodOrder(MethodSorters.NAME_ASCENDING): Sorts the test methods by method name, in lexicographic order.

Code:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class ApplicationTest extends ActivityInstrumentationTestCase2<MainActivity> {

    public ApplicationTest() {
        super(MainActivity.class);
    }

    @Rule
    public ActivityTestRule<MainActivity> mActivityTestRule = new ActivityTestRule<>(MainActivity.class);

    @Test
    void t1AttachUI(){
        // testing code goes here
    }

    @Test
    void t2InitializeViews(){
        // testing code goes here
    };

    @Test
    void t3SettingValues(){
        // testing code goes here
    };

    @Test
    void t4Validation(){
        // testing code goes here
    };

    @Test
    void t3AfterButtonPress(){
        // testing code goes here
    };
}

Upvotes: 7

pdmoore
pdmoore

Reputation: 76

Unit tests ought to be independent so most frameworks don't guarantee or enforce the order in which they are run. But since you want to enforce an order the easiest way I've done it in the past it to create a "throw away" test suite or test method that calls the tests in whatever order I want them to run in. Unit tests are methods, just call them. This is easy to do if you're dealing with tens of tests, not at all appealing if you're dealing with hundreds or thousands.

Try to isolate the flaky interaction as much as possible, then swap around the order of the poorly interacting tests within the throwaway calling method.

Upvotes: 3

Related Questions