Gradle Jacoco report coverage percent doesn't match verification percent

Question

We configured excludes:

def coverageExcludes() {
    return [
        "com/ourcompany/*Config*",
        "com/ourcompany/ServiceApplication*",
        "com/ourcompany/Shutdown*",
        "com/ourcompany/ShutdownConnector*",
        "com/ourcompany/Tomcat*",
        "com/ourcompany/TomcatCustomConnectorCustomizer*",
        "com/ourcompany/endpoint/exception/**",
        "com/ourcompany/endpoint/util/ObjectMapperBuilder*",
        "com/ourcompany/framework/**",
        "com/ourcompany/helper/ApiHelper*",
        "com/ourcompany/helper/OffsetDateTimeDeserializer*",
        "com/ourcompany/persistence/entity/**",
        "com/ourcompany/persistence/exception/**",
        "com/ourcompany/service/exception/ServiceException*",
        "com/ourcompany/service/model/**",
        "com/ourcompany/v2/**",
        "com/ourcompany/v3/**"
    ]
}

used them in jacocoTestReport:

jacocoTestReport {
    reports {
        xml {
            enabled true
        }
        html {
            enabled true
        }
    }
    getExecutionData().setFrom(fileTree(buildDir).include("/jacoco/*.exec"));
    afterEvaluate {
        classDirectories.setFrom(files(classDirectories.files.collect {
            fileTree(dir: it,
                exclude: coverageExcludes()
            )
        }))
        for (Object c : classDirectories) {
            System.out.println(c);
        }
    }
}

and the excludes work:

Then we tried to use the same excludes in the verification task, with the coverage-failure threshold set to 0.87 because that's what is displayed in the coverage report:

jacocoTestCoverageVerification {
    afterEvaluate {
        classDirectories.setFrom(files(classDirectories.files.collect {
            fileTree(dir: it,
                exclude: coverageExcludes()
            )
        }))
        for (Object c : classDirectories) {
            System.out.println(c);
        }
    }
    violationRules {
        rule {
            element = 'BUNDLE'
            limit {
                counter = 'INSTRUCTION'
                value = 'COVEREDRATIO'
                minimum = 0.87
            }
        }
    }
}
check.dependsOn jacocoTestCoverageVerification

But that doesn't work:

Execution failed for task ':settings:jacocoTestCoverageVerification'.

Rule violated for bundle settings: instructions covered ratio is 0.70, but expected minimum is 0.87

We compared the output of classDirectories in both tasks, and they match exactly, and nothing in the list of excludes is in the classDirectories. Which means both tasks see the same classes.

Yet the verification task gets a different answer on coverage than the report task.

This is worrisome, because we don't know which task is getting the right answer. If it's the report task, we can just live with the lower threshold in the verification task, but if we really do only have 70% coverage, we'd like to know that.

As near as we can tell from the documentation, we have configured the verification task to analyze the coverage in the exact same way that the report task generates.

What are we doing wrong?

(We're using Gradle 5.5.1, and can't upgrade yet, but will if we can get past this problem.)

Answer 1

You configure ExecutionData in your jacocoTestReport task

getExecutionData().setFrom(fileTree(buildDir).include("/jacoco/*.exec"));

but not in jacocoTestCoverageVerification task.