Tests do not run in the specified order in JUnit5

Question

I am new to Kotlin, Java and IntelliJ and I have been pulling my hair since the past few hours and i need some help.

I need my tests to run in a specific order. I know that it is not recommended, but these are integration and workflow tests and not unit tests. For some reason, JUnit 5 totally ignored the test order in IntelliJ and runs the tests in a random order. I wrote a simple piece of code to check this and the behaviour is totally random.

import org.junit.jupiter.api.MethodOrderer.OrderAnnotation
import org.junit.jupiter.api.Order
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.TestMethodOrder

@TestMethodOrder(OrderAnnotation::class)
class OrderedTests
{

    @Test
    @Order(1)
    fun `A Test`()
    {
        println("First Test")
    }

    @Test
    @Order(2)
    fun `A Second Test`()
    {
        println("Second Test")
    }

    @Test
    @Order(3)
    fun `A FFFF Test`()
    {
        println("Third Test")
    }

    @Test
    @Order(4)
    fun `1 Test`()
    {
        println("Fourth Test")
    }

    @Test
    @Order(5)
    fun `AA Test`()
    {
        println("Fifth Test")
    }

    @Test
    @Order(6)
    fun `MM Test`()
    {
        println("Sixth Test")
    }
}

Here is how my pom.xml dependencies look like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>de.arvatoinfoscore.lab.ad.wf</groupId>
    <artifactId>dummytest</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <version.java>11.0.4</version.java>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
        <junit.jupiter.version>5.5.2</junit.jupiter.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <kotlin.version>1.3.50</kotlin.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-stdlib-jdk8</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-test-junit5</artifactId>
            <version>${kotlin.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <finalName>${project.artifactId}</finalName>
        <plugins>
            <plugin>
                <groupId>org.jetbrains.kotlin</groupId>
                <artifactId>kotlin-maven-plugin</artifactId>
                <version>${kotlin.version}</version>
                <executions>
                    <execution>
                        <id>compile</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>test-compile</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>test-compile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <jvmTarget>1.8</jvmTarget>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

I am using IntelliJ to run my tests and the run order is always the same: Fifth Test, Second Test, Third Test, Fourth Test, Sixth Test and First Test.

Any help to point me in the right direction will be greatly appreciated. Thanks

Answer 1

You need junit-jupiter-engine as a dependency. Add this to your pom:

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>${junit.jupiter.version}</version>
    <scope>test</scope>
</dependency>

Answer 2

As you said order should be irrelevant, take a look a this https://github.com/junit-team/junit5/issues/48