CODEWITHSUNDEEP
automationswaggerpacttest-coverageenunciate
melli79
melli79

Reputation: 21

measure pact test coverage of an API

Given the importance of automated tests I am wondering if there are possibilities to measure the test coverage for Pact tests.

Remember that for unit tests, most frameworks/IDEs provide means to check which parts of the code were executed and thus to check if at least every part of the code (say in a class) was executed by some test.

Correspondingly, a Pact test (that intends to check the conformance of a provider or consumer to an API specification) gains credibility by testing most of the possible interactions. If it does, then it can reduce the number of e2e tests by splitting their logic into a functional part, covered by unit and integration tests within one component, and pact tests that assert the conformity of both components to the agreed upon API. (See https://docs.pact.io/getting_started/what_is_pact_good_for/ for more details.)

Given that it is possible to automatically discover the REST API of some web components via libraries like enunciate or swagger, I am wondering whether it is possible to compare their findings with the PACT test descriptions to see whether any important parameters or endpoints are missing in the latter. (Also, I am not sure if it means doubling the efforts to provide enunciate annotations and a PACT contract, but it seemed to me that the enunciate annotations in Java are not that heavy, i.e. mostly identifying the REST endpoints by the REST annotations that are necessary for functionality anyway.)

Please let me know if my question is too general for stackoverflow and I should rather ask it on more academic sites like stackexchange.

Upvotes: 1

Views: 1232

Answers (1)

Beth Skurrie
Beth Skurrie

Reputation: 1318

From the docs.pact.io question archive:

How can I tell if I have good contract test coverage of my API?

This is actually the wrong question to be asking. Contract tests aren't intended to provide any particular percentage coverage of the provider (that's what the provider's own functional tests are for). Contract tests are meant to provide (as close to) 100% coverage of the consumer code that makes the calls to the provider (you can think of this as the "provider client code"). If you execute your consumer Pact tests in a separate step in your test suite, you can use standard code coverage tools to determine whether or not your Pact tests have covered a sufficient percentage of your provider client code.

https://docs.pact.io/faq/question_archive#how-can-i-tell-if-i-have-good-contract-test-coverage-of-my-api

This doesn't tell you whether you've covered all the different variations of the parameters unfortunately. It's often not completely feasible to ensure you have a contract test for every single parameter variation (imagine having to try and cover 20 different combinations of a particular enum). At that stage, it can be a more efficient option to put in a number of well chosen test cases, and then make sure your code can gracefully handle any exceptions (eg. have good monitoring that reports unexpected values back to the dev team).

Upvotes: 2

Related Questions