CODEWITHSUNDEEP
scalasbtscalatest
Rogach
Rogach

Reputation: 27200

Why does scalatest mix up the output?

I run my scalatest from sbt, and the output gets mixed up - scalatest prints all the test run, and comments to them, and somewhere in the middle it prints the statistics:

> test
[info] Compiling 1 Scala source to /home/platon/Tor/scala-dojo-02/target/scala-2.9.1/classes...
[info] FunsWithListsTests:
[info] - should return list of labels
[info] - should return the average rating of games belonging to Zenga
[info] - should return the total ratings of all games
[info] - should return the total ratings of EA games *** FAILED ***
[info]   0 did not equal 170 (FunsWithListsTests.scala:35)
[error] Failed: : Total 8, Failed 5, Errors 0, Passed 3, Skipped 0
[info] - should increase all games rating by 10 *** FAILED ***
[error] Failed tests:
[error]     dojo.FunsWithListsTests
[info]   List() did not equal List(Game(Activision,40), Game(Zenga,70), Game(Zenga,20), Game(EA,70), Game(EA,120)) (FunsWithListsTests.scala:40)
[info] - should decrease all Zenga games rating by 10 *** FAILED ***
[info]   List() did not equal List(Game(Activision,30), Game(Zenga,50), Game(Zenga,0), Game(EA,60), Game(EA,110)) (FunsWithListsTests.scala:45)
[info] - should create function to find Activision games *** FAILED ***
[info]   List(Game(Activision,30), Game(Zenga,60), Game(Zenga,10), Game(EA,60), Game(EA,110)) did not equal List(Game(Activision,30)) (FunsWithListsTests.scala:50)
[info] - should return a List of tuples consisting of game label and game *** FAILED ***
[info]   List() did not equal List((ACTIVISION,Game(Activision,30)), (ZENGA,Game(Zenga,60)), (ZENGA,Game(Zenga,10)), (EA,Game(EA,60)), (EA,Game(EA,110))) (FunsWithListsTests.scala:56)
[error] {file:/home/platon/Tor/scala-dojo-02/}default-940f03/test:test: Tests unsuccessful
[error] Total time: 1 s, completed Mar 20, 2012 9:27:13 AM

It seems that if I would accumulate a great number of tests, searching for those stats and failed tests would become a pain.

Is there a way to fix this?

Upvotes: 14

Views: 1243

Answers (2)

Edmondo
Edmondo

Reputation: 20080

It looks to me the reason for that is that SBT by default executes the tests in parallel, the same way maven-surefire-plugin does.

As it is explained in ScalaTest wiki, this can be solved by:

Disable Parallel Execution of Tests By default, sbt runs all tasks in parallel. Because each test is mapped to a task, tests are also run in parallel by default. To disable parallel execution of tests:

parallelExecution in Test := false

Upvotes: 7

arz.freezy
arz.freezy

Reputation: 688

Have the same issue and solved it by saving logs into separate files. Not ideal, but good for CI, especially in case of long-running integration tests. How I did it:

  1. Create a special directory for logs (for example, at the end of build.sbt):

    lazy val logDirectory = taskKey[File]("A directory for logs")
logDirectory := {
  val r = target.value / "logs"
  IO.createDirectory(r)
  r
}

  2. [Optionally] Specify a file for a summary information in a project:

    testOptions in test += Tests.Argument(
  TestFrameworks.ScalaTest, "-fW", (logDirectory.value / "summary.log").toString
)

    W disables colors

  3. Create a group for each test. Each group must run a test in a forked JVM with special arguments:

    testGrouping in test := testGrouping.value.flatMap { group =>
  group.tests.map { suite =>
    val fileName = {
      // foo.bar.baz.TestSuite -> f.b.b.TestSuite
      val parts = suite.name.split('.')
      (parts.init.map(_.substring(0, 1)) :+ parts.last).mkString(".")
    }

    val forkOptions = ForkOptions(
      bootJars = Nil,
      javaHome = javaHome.value,
      connectInput = connectInput.value,
      outputStrategy = outputStrategy.value,
      runJVMOptions = javaOptions.value ++ Seq(
        "-Dour.logging.appender=FILE",
        s"-Dour.logging.dir=${logDirectory.value / fileName}"
      ),
      workingDirectory = Some(baseDirectory.value),
      envVars = envVars.value
    )

    group.copy(
      name = suite.name,
      runPolicy = Tests.SubProcess(forkOptions),
      tests = Seq(suite)
    )
  }
}

    Note, we can tell the logback, where we save our logs through our.logging.appender and our.logging.dir

  4. Create a configuration for logging (test/resources/logback-test.xml):

    <?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <target>System.out</target>
        <encoder>
            <pattern>%date{yyyy-MM-dd HH:mm:ss} %-5level [%thread] %logger{26} - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.FileAppender">
        <file>${our.logging.dir}/test.log</file>
        <append>false</append>
        <encoder>
            <pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] %logger{26} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="TRACE">
        <appender-ref ref="${our.logging.appender}"/>
    </root>
</configuration>

That's all.

Upvotes: 0

Related Questions