CODEWITHSUNDEEP
testinggroovyautomated-testsspock
testybanana
testybanana

Reputation: 3

Executing all assertions in the same Spock test, even if one of them fails

I am trying to verify two different outputs in the context of a single Spock method that runs multiple test cases of the form when-then-where. For this reason I use two assertions at the then block, as can be seen in the following example:

import spock.lang.*

@Unroll
class ExampleSpec extends Specification {

     def "Authentication test with empty credentials"() {
         when:
         def reportedErrorMessage, reportedErrorCode
         (reportedErrorMessage, reportedErrorCode) = userAuthentication(name, password)

         then:
         reportedErrorMessage == expectedErrorMessage
         reportedErrorCode    == expectedErrorCode

         where:
         name | password || expectedErrorMessage | expectedErrorCode
         ' '  | null     || 'Empty credentials!' | 10003
         ' '  | ' '      || 'Empty credentials!' | 10003
     }
}

The code is an example where the design requirement is that if name and password are ' ' or null, then I should always expect exactly the same expectedErrorMessage = 'Empty credentials!' and expectedErrorCode = 10003. If for some reason (presumably because of bugs in the source code) I get expectedErrorMessage = Empty! (or anything else other than 'Empty credentials!') and expectedErrorCode = 10001 (or anything else other than 1003), this would not satisfy the above requirement.

The problem is that if both assertions fail in the same test, I get a failing message only for the first assertion (here for reportedErrorMessage). Is it possible to get informed for all failed assertions in the same test?

Here is a piece of code that demonstrates the same problem without other external code dependencies. I understand that in this particular case it is not a good practice to bundle two very different tests together, but I think it still demonstrates the problem.

import spock.lang.*

@Unroll
class ExampleSpec extends Specification {

    def "minimum of #a and #b is #c and maximum of #a and #b is #d"() {
        expect:
        Math.min(a, b) == c
        Math.max(a, b) == d

        where:
        a | b || c | d
        3 | 7 || 3 | 7
        5 | 4 || 5 | 4  // <--- both c and d fail here
        9 | 9 || 9 | 9
    }
}

Upvotes: 0

Views: 2939

Answers (2)

mnd
mnd

Reputation: 2789

Based on the latest comment by OP, it looks like a solution different from my previous answer would be helpful. I'm leaving the previous answer in-place, as I feel it still provides useful information related to the question (specifically separating positive and negative tests).

Given that you want to see all failures, and not just have it fail at the first assert that fails, I would suggest concatenating everything together into a boolean AND operation. Not using the && shortcut operator, because it will only run until the first check that does not satisfy the entire operation. I would suggest using the &, so that all checks are made, regardless of any previously failing checks.

Given the max and min example above, I would change the expect block to this:

Math.min(a, b) == c & Math.max(a, b) == d

When the failure occurs, it gives you the following information:

Math.min(a, b) == c & Math.max(a, b) == d
     |   |  |  |  | |      |   |  |  |  |
     4   5  4  |  5 false  5   5  4  |  4
               false                 false

This shows you every portion of the failing assert. By contrast, if you used the &&, it would only show you the first failure, which would look like this:

Math.min(a, b) == c && Math.max(a, b) == d
     |   |  |  |  | |
     4   5  4  |  5 false
               false

This could obviously get messy pretty fast if you have more than two checks on a single line - but that is a tradeoff you can make between all failing information on one line, versus having to re-run the test after fixing each individual component.

Hope this helps!

Upvotes: 2

mnd
mnd

Reputation: 2789

I think there are two different things at play here.

  1. Having a failing assert in your code will throw an error, which will cease execution of the code. This is why you can't have two failing assertions in a single test. Any line of code in an expect or then block in Spock has an implicit assert before it.
  2. You are mixing positive and negative unit tests in the same test. I ran into this before myself, and I read/watched something about this and Spock (I believe from the creator, Peter Niederwieser), and learned that these should be separated into different tests. Unfortunately I couldn't find that reference. So basically, you'll need one test for failing use cases, and one test for passing/successful use cases.

Given that information, here is your second example code, with the tests separated out, with the failing scenario in the second test.

    @Unroll
    class ExampleSpec extends Specification {

        def "minimum of #a and #b is #c and maximum of #a and #b is #d - successes"() {
            expect:
            Math.min(a, b) == c
            Math.max(a, b) == d

            where:
            a | b || c | d
            3 | 7 || 3 | 7
            9 | 9 || 9 | 9
        }

        def "minimum of #a and #b is #c and maximum of #a and #b is #d - failures"() {
            expect:
            Math.min(a, b) != c
            Math.max(a, b) != d

            where:
            a | b || c | d
            5 | 4 || 5 | 4
        }
    }

As far as your comment about the MongoDB test case - I'm not sure what the intent is there, but I'm guessing they are making several assertions that are all passing, rather than validating that something is failing.

Upvotes: 1

Related Questions