CODEWITHSUNDEEP
scalasbt
som-snytt
som-snytt

Reputation: 39577

How do I sbt test continually until a flakey test fails?

If there is a test that fails only sometimes, can I ask sbt to run tests continually until failure?

Otherwise, I'm stuck hitting up-arrow while watching Arrow. (pun unintended but leavening)

https://stackoverflow.com/q/22366719/1296806

Upvotes: 10

Views: 3340

Answers (4)

KrzyH
KrzyH

Reputation: 4326

I have created sbt plugin for flaky test detection: sbt-flaky. You can run test for:

  • specified duration sbt clean "flaky duration=30",
  • specified times sbt clean "flaky times=30"
  • until first failure sbt clean "flaky firstFail".

Advantage of this plugin is aggregation of failures, history trends and possibility of adding flaky test detection into pipeline.

Upvotes: 1

lpiepiora
lpiepiora

Reputation: 13749

I had the same problem, and I've ended up implementing this as a command.

lazy val root = Project("test", file(".")).
  settings(commands += Command.command("testUntilFailed") { state =>
    "test" :: "testUntilFailed" :: state
  })

The advantage is that it will skip the SBT loading. You can also add extra parameter or run testOnly to test a single test.

Upvotes: 23

johanandren
johanandren

Reputation: 11479

Old question, but having just had the same need myself, here is a solution: sbt will return non-zero exit code if you run it one off with the tests, so, one way to loop until it fails is to just look at the exit code in the shell:

while [ $? -eq 0 ]; do sbt test; done

Upvotes: 9

Jacek Laskowski
Jacek Laskowski

Reputation: 74709

In my opinion it doesn't particularly apply to SBT or any other build management tool. There's no built-in SBT feature for this. You'd have to build one, but it'd be far from what SBT is meant to offer - build configuration management. It can take quite a while to hit the case which causes the build/test fail. It's very unpredictable.

When your test fails, it means that there are cases the test doesn't pass. The error should tell you what they are that you use to improve the test.

ScalaCheck: Property-based testing for Scala might be of some help.

Upvotes: 0

Related Questions