CODEWITHSUNDEEP
groovyspock
rdmueller
rdmueller

Reputation: 11042

Is it possible to randomize the order in which Spock tests are executed?

it seems that spock tests are executed in the same order, most of the time.

Is it possible to set some option to execute them in a random order?

Update: as tim_yates commented "tests should be isolated, and order shouldn't matter", I think I should explain why I would like to have this feature...

We had a code retreat where we tried to just tried to turn the tests green. So we implemented a state in the class under test which then would be used to return the corerct result for all tests.

To avoid such evil coding, I thought it would be great to execute tests in random order :-)

Upvotes: 2

Views: 552

Answers (2)

Szymon Stepniak
Szymon Stepniak

Reputation: 42234

After Leonard Brünings suggestion, I've replaced the solution based on extending Sputnik with using annotation-driven extension.

You can create your own Spock extension that randomizes features. Consider the following example.

package com.github.wololock

import spock.lang.Specification

@RandomizedOrder
class RandomSpockSpec extends Specification {

    def "test 1"() {
        when:
        def number = 1

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 2"() {
        when:
        def number = 2

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 3"() {
        when:
        def number = 3

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 4"() {
        when:
        def number = 4

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 5"() {
        when:
        def number = 5

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }
}

src/test/groovy/com/github/wololock/RandomSpockSpec.groovy

This specification contains 5 features that prints numbers to the console. We use a custom @RandomizeOrder annotation (see "Annotation-Driven Local Extension" docs). Firstly, we create an annotation class.

package com.github.wololock

import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension
import org.spockframework.runtime.model.SpecInfo

final class RandomizedOrderExtension extends AbstractAnnotationDrivenExtension<RandomizedOrder> {

    public static final String SPOCK_RANDOM_ORDER_SEED = "spock.random.order.seed"

    private static final long seed = System.getProperty(SPOCK_RANDOM_ORDER_SEED)?.toLong() ?: System.nanoTime()

    static {
        println "Random seed used: ${seed}\nYou can re-run the test with predefined seed by passing -D${SPOCK_RANDOM_ORDER_SEED}=${seed}\n\n"
    }

    @Override
    void visitSpecAnnotation(RandomizedOrder annotation, SpecInfo spec) {
        final Random random = new Random(seed)

        final List<Integer> order = (0..(spec.features.size())) as ArrayList

        Collections.shuffle(order, random)

        spec.features.each { feature ->
            feature.executionOrder = order.pop()
        }
    }
}

src/test/groovy/com/github/wololock/RandomizedOrderExtension.groovy

This extension does one thing - in the visitSpec visitor method we assign a random execution order to all feature methods. It supports predefined seed, so whenever you want to re-create a specific order, you can read the seed value from the console and you can pass it in the next run. For instance, the following parameter added -Dspock.random.order.seed=1618636504276 will shuffle features using predefined seed.

When we run the test annotated with @RandomizedOrder we will see methods in the order different than the declaration order.

enter image description here

Upvotes: 6

Leonard Br&#252;nings
Leonard Br&#252;nings

Reputation: 13242

As you only want to randomize the order of tests inside a class, then this can be achieved with an extension. Take a look at StepwiseExtension 

private void sortFeaturesInDeclarationOrder(SpecInfo spec) {
    for (FeatureInfo feature : spec.getFeatures())
      feature.setExecutionOrder(feature.getDeclarationOrder());
 }

You can simply randomize the execution order, in you own extension. Depending on whether you want it only selectively or for all your tests. You can create either an AnnotationDrivenExtension or a IGlobalExtension see the docs on how that works.

Upvotes: 5

Related Questions