CODEWITHSUNDEEP
scalascalacheckproperty-based-testing
Andrzej Wąsowski
Andrzej Wąsowski

Reputation: 674

Negating the outcome of scalacheck tests (properties)

I am using scalacheck and would like to evaluate the same set of properties on two functions f1 and f2, but I would like to interpret the outcomes differently. For f1 I would like the properties to pass, while for f2 I would like the properties to fail. This is needed for simple mutation testing (to check whether the test is good enough to detect bugs).

I have put my tests in a class, which I parameterize with the function to test. I made the class abstract to prevent the test runner from running this class (otherwise it fails with an error that there is no nullary constructor).

abstract class MySpec(f: SOMETYPE, suffix: String) 
    extends org.scalacheck.Properties(s"MySpec.${suffix}"): 

    property("trivial") = true

Now in a test file I would like to instantiate it twice. Something like this:

object CorrectSpec extends MySpec(f1, "correct")
abstract class MutantSpec extends MySpec(f2, "mutant")

The second spec is abstract to prevent it from executing before I negate the properties. Unfortunately, I could not figure out how to do this, at least not without a major change to how MySpec is written, which I would like to avoid - this is code facing my users, and I would like to hide what I am doing behind the scenes.

I tried various things, but they all end up in one of two corners. I can extract properties using MySpec(f2, "mutant").properties and then map or flatMap to create the negated ones. For example with flatMap (just for the first property, for simplicity):

mutantSpec.properties(0).flatMap { result => result.status == False }

Now that I have the negated properties, I need to add them to some spec objects for execution.

(A) I can inject them in MutantSpec (extending MutantSpec and using property.update). Sadly, I am getting an error that updating properties during execution is not allowed. This is a strange message, as this is not more "during execution" than the other properties defined in the same file. This seems to be caused by the properties methods that I use to access the existing "correct" properties. It seems to set a flag that the values have leaked (so further updates are not allowed).

(B) I can also inject the properties into a new spec that does not extend MutantSpec. Then the error does not appear (as I am modifing another spec object than the one "frozen" by accessing properties. But in order to call properties from outside MutantSpec I have to instantiate MutantSpec which makes scala-cli execute the unmodfied MutantSpec and by-design this spec will fail (as it is the non-negated spec, running on a buggy f2). This pollutes my test results and breaks CI.

Any idea? Did I get into a corner scalacheck's designers have not anticipated? The second idea (B) seems closer as it runs, just slightly too many tests. Perhaps I can filter out the annoying tests with a command line option, but as far as I can see only a positive filter (-f) is available. I worked around this by formatting my messages so all the actual tests contain a selected string, and those that are spurious do not. It is not ideal, as it limits the names of tests, but I could not figure out a better way.

Upvotes: 1

Views: 50

Answers (0)

Related Questions