ScalaCheck shrinking command data in stateful testing

Question

When doing stateful testing with ScalaCheck the library can shrink the commands needed to find a certain bug. Like in the counter example from the userguide: https://github.com/typelevel/scalacheck/blob/master/doc/UserGuide.md. But what if the commands took arguments and i wanted ScalaCheck to shrink the data inside the commands as well? See the scenario where i am testing the Counter below:

package Counter

case class Counter() {
  private var n = 1
  def increment(incrementAmount: Int) = {
    if (n%100!=0) {
      n += incrementAmount
    }
  }
  def get(): Int = n
}

The counter is programmed with a bug. It should not increment with the given amount if n%100 == 0. So if the value of n is x*100, where x is any positive integer, the counter is not incremented. I am testing the counter with the ScalaCheck stateful test below:

import Counter.Counter

import org.scalacheck.commands.Commands
import org.scalacheck.{Gen, Prop}
import scala.util.{Success, Try}

object CounterCommands extends Commands {
  type State = Int
  type Sut = Counter

  def canCreateNewSut(newState: State, initSuts: Traversable[State],
                      runningSuts: Traversable[Sut]): Boolean = true
  def newSut(state: State): Sut = new Counter
  def destroySut(sut: Sut): Unit = ()
  def initialPreCondition(state: State): Boolean = true
  def genInitialState: Gen[State] = Gen.const(1)
  def genCommand(state: State): Gen[Command] = Gen.oneOf(Increment(Gen.chooseNum(1, 200000).sample.get), Get)

  case class Increment(incrementAmount: Int) extends UnitCommand {
    def run(counter: Sut) = counter.increment(incrementAmount)
    def nextState(state: State): State = {state+incrementAmount}
    def preCondition(state: State): Boolean = true
    def postCondition(state: State, success: Boolean) = success
  }

  case object Get extends Command {
    type Result = Int
    def run(counter: Sut): Result = counter.get()
    def nextState(state: State): State = state
    def preCondition(state: State): Boolean = true
    def postCondition(state: State, result: Try[Int]): Prop = result == Success(state)
  }
}

Everytime the increment command is chosen it is given some arbitrary integer between 1 and 200000 as argument. Running the test gave the following output:

! Falsified after 28 passed tests.
> Labels of failing property:
initialstate = 1
seqcmds = (Increment(1); Increment(109366); Increment(1); Increment(1); Inc
  rement(104970); Increment(27214); Increment(197045); Increment(1); Increm
  ent(54892); Get => 438600)
> ARG_0: Actions(1,List(Increment(1), Increment(109366), Increment(1), Incr
  ement(1), Increment(104970), Increment(27214), Increment(197045), Increme
  nt(1), Increment(54892), Get),List())
> ARG_0_ORIGINAL: Actions(1,List(Get, Get, Increment(1), Increment(109366),
   Get, Get, Get, Get, Increment(1), Get, Increment(1), Increment(104970),
  Increment(27214), Get, Increment(197045), Increment(1), Increment(54892),
   Get, Get, Get, Get, Get, Increment(172491), Get, Increment(6513), Get, I
  ncrement(57501), Increment(200000)),List())

ScalaCheck did indeed shrink the commands needed to find the bug (as can be seen in ARG_0) but it did not shrink the data inside the commands. It ended up with a much larger counter value (438600) than what is actually needed to find the bug. If the first increment command was given 99 as argument the bug would be found.

Is there any way in ScalaCheck to shrink the data inside the commands when running stateful tests? The ScalaCheck version used is 1.14.1.

EDIT: I tried simplifying the bug (and only incrementing if n!=10) and added the shrinker that was suggested by Levi and could still not get it to work. The whole runnable code can be seen below:

package LocalCounter

import org.scalacheck.commands.Commands
import org.scalacheck.{Gen, Prop, Properties, Shrink}
import scala.util.{Success, Try}

case class Counter() {
  private var n = 1
  def increment(incrementAmount: Int) = {
    if (n!=10) {
      n += incrementAmount
    }
  }
  def get(): Int = n
}

object CounterCommands extends Commands {
  type State = Int
  type Sut = Counter

  def canCreateNewSut(newState: State, initSuts: Traversable[State],
                      runningSuts: Traversable[Sut]): Boolean = true
  def newSut(state: State): Sut = new Counter
  def destroySut(sut: Sut): Unit = ()
  def initialPreCondition(state: State): Boolean = true
  def genInitialState: Gen[State] = Gen.const(1)
  def genCommand(state: State): Gen[Command] = Gen.oneOf(Increment(Gen.chooseNum(1, 40).sample.get), Get)

  case class Increment(incrementAmount: Int) extends UnitCommand {
    def run(counter: Sut) = counter.increment(incrementAmount)
    def nextState(state: State): State = {state+incrementAmount}
    def preCondition(state: State): Boolean = true
    def postCondition(state: State, success: Boolean) = success
  }

  case object Get extends Command {
    type Result = Int
    def run(counter: Sut): Result = counter.get()
    def nextState(state: State): State = state
    def preCondition(state: State): Boolean = true
    def postCondition(state: State, result: Try[Int]): Prop = result == Success(state)
  }

  implicit val shrinkCommand: Shrink[Command] = Shrink({
      case Increment(amt) => Shrink.shrink(amt).map(Increment(_))
      case Get => Stream.empty
  })
}

object CounterCommandsTest extends Properties("CounterCommands") {
  CounterCommands.property().check()
}

Running the code gave the following output:

! Falsified after 4 passed tests.
> Labels of failing property:
initialstate = 1
seqcmds = (Increment(9); Increment(40); Get => 10)
> ARG_0: Actions(1,List(Increment(9), Increment(40), Get),List())
> ARG_0_ORIGINAL: Actions(1,List(Increment(9), Increment(34), Increment(40)
  , Get),List())

Which is not the minimal example.

Answer 1

I've been trying to push for more flexible shrinking here: https://github.com/typelevel/scalacheck/pull/740/files

Here's maybe one idea:

• Make a copy of the Commands.scala file from scalacheck in your own project
• Alter it to make the Actions case class and shrinkActions field protected
• Override shrinkActions in your test cases

Since shrinkActions is the shrinker that shrinks everything, you can put in your own command and state shrinkers as you please.

Answer 2

You should be able to define a custom Shrink for Command along these lines:

implicit val shrinkCommand: Shrink[Command] = Shrink({
  case Increment(amt) => shrink(amt).map(Increment(_))
  case Get => Stream.empty
}

Note that, because Stream is deprecated in Scala 2.13, you may need to disable warnings in Scala 2.13 (Scalacheck 1.15 will allow LazyList to be used to define shrinks).