CODEWITHSUNDEEP
rubyminitest
Petr Skocik
Petr Skocik

Reputation: 60097

Minitest i_suck_and_my_tests_are_order_dependent

I'm using the spec syntax of minitest with describe blocks. In my minitest_helper.rb (what each spec file requires) I have:

Minitest::Test.i_suck_and_my_tests_are_order_dependent!

This makes the test within a describe block run in order. Still, the order of describe blocks is random.

How do I make everything in-order?

Demonstrating the problem:

require 'minitest/autorun'
require 'minitest/documentation'
Minitest::Test.i_suck_and_my_tests_are_order_dependent!

describe "Block1" do

    3.times.each do |i|
      it "should print #{i+1}"
    end
end
describe "Block2" do

    3.times.each do |i|
      it "should print #{i+1}"
    end
end

I expect it to output:

Block1
  0001 should print 1
  0002 should print 2
  0003 should print 3
Block2
  0001 should print 1
  0002 should print 2
  0003 should print 3

But sometimes Block2 comes first.Adding more blocks shows that the blocks are generally run in random order.

What do I want this for?

Documentation

Randomizing tests is great for making sure tests are independent as they should be (in other words, it's great for debugging the test suite). After I'm done debugging the test suite, though, I want the test-suite to serve as documentation, and documentation written in random order kind of sucks.

Upvotes: 2

Views: 2379

Answers (2)

Christian Nadeau
Christian Nadeau

Reputation: 61

For those who are still looking for a way to do this, put this in your test_helper.rb

# disable random test order for any TestCase
ActiveSupport::TestCase.class_eval do
  # http://docs.seattlerb.org/minitest/Minitest/Test.html#method-c-i_suck_and_my_tests_are_order_dependent-21
  i_suck_and_my_tests_are_order_dependent!
end

MiniTest.remove_possible_singleton_method(:__run)
Minitest.define_singleton_method(:__run) do  |reporter, options|
  suites = Minitest::Runnable.runnables.reject { |s| s.runnable_methods.empty? }
  parallel, serial = suites.partition { |s| s.test_order == :parallel }

  # If we run the parallel tests before the serial tests, the parallel tests
  # could run in parallel with the serial tests. This would be bad because
  # the serial tests won't lock around Reporter#record. Run the serial tests
  # first, so that after they complete, the parallel tests will lock when
  # recording results.
  serial.map { |suite| suite.run reporter, options } +
    parallel.map { |suite| suite.run reporter, options }
end

From: https://www.gregnavis.com/articles/how-to-reduce-test-interference-in-minitest.html

Upvotes: 3

Petr Skocik
Petr Skocik

Reputation: 60097

It doesn't seem like there was an option to turn this off, so I added it: https://github.com/seattlerb/minitest/pull/550

Minitest.should_shuffle_suites = false

should now turn it off with the minitest from my fork.

Upvotes: 2

Related Questions