F. Zer
F. Zer

Reputation: 1279

How can I pinpoint the specific location where property tests fail when using QuickCheck more efficiently?

Current solution:

I am currently using counterexample function from Test.Quickcheck:

counterexample :: Testable prop => String -> prop -> Property

to add informative messages in order to know exactly where the test failed. However, let's say I have multiple properties inside a property:

f s = 
    counterexample "p1 s" (p1 s) .&&.
    counterexample "p2 s" (p2 s) .&&.
    counterexample "p3 s" (p3 s) .&&.
    counterexample "p4 s" (p4 s) .&&.
    counterexample "p5 s" (p5 s)

where p1...p5 have type Property.

This hinders readability. I would love to have something like:

prop s = 
    ce (p1 s) .&&.
    ce (p2 s) .&&.
    ce (p3 s) .&&.
    ce (p4 s) .&&.
    ce (p5 s)

ce (p1 s)

would expand to:

counterexample "p1 s" (p1 s)

Is there any solution using QuickCheck library, TemplateHaskell or something else? Would be great to know automatically which expression failed a test without explicitly using counterexample function and writing the expression that is on the right side.

EDIT:

In order to clarify my question, I am adding a minimal working example:

{-# LANGUAGE TemplateHaskell #-}

module Main where

import Test.QuickCheck
import System.Exit

-- Property 1: Reversing a list twice should give the original list.
propertyReverseTwice :: [Int] -> Bool
propertyReverseTwice xs = reverse (reverse xs) == xs

-- Property 2: The length of the reversed list should be the same as the original list.
propertyReverseLengthFail :: [Int] -> Bool
propertyReverseLengthFail xs = length (reverse xs) == length xs + 1

-- Property 3: Combining the first two properties.
prop_combined :: [Int] -> Property
prop_combined xs = 
  propertyReverseTwice xs .&&. propertyReverseLengthFail xs

return []

main :: IO ()
main = do
  passed <- $quickCheckAll
  exitWith $ if passed
    then ExitSuccess
    else ExitFailure 1

I am adding (on purpose) a property that never holds (property 2). Running this code gives:

=== prop_combined from /Users/detach/Haskell-Foundations/Recursion/tempCodeRunnerFile.haskell:17 ===
*** Failed! Falsified (after 1 test):  
[]

I see line 17 is where the error occurs. However, I don't know which of the properties failed. On the other hand, if I modify the code to include counterexample function:

-- Property 3: Combining the first two properties.
prop_combined :: [Int] -> Property
prop_combined xs = 
  counterexample ("propertyReverseTwice " <> show xs) (propertyReverseTwice xs) .&&.
  counterexample ("propertyReverseLengthFail " <> show xs) (propertyReverseLengthFail xs)

The error given is more informative (I know which property failed

=== prop_combined from /Users/detach/Haskell-Foundations/Recursion/tempCodeRunnerFile.haskell:17 ===
*** Failed! Falsified (after 1 test):  
[]
propertyReverseLengthFail []

So, how can I avoid writing, for example:

counterexample ("propertyReverseTwice " <> show xs)

every time, and instead use a shorthand using TemplateHaskell or some other mechanism ?

Upvotes: 1

Views: 126

Answers (0)

Related Questions