NodeJS Code Coverage of Automated Tests

Question

As part of a custom testing framework for a NodeJS REST API, I'd like to automatically detect when my tests are no longer providing proper coverage by comparing all possible outcomes with what the test suite received.

What methods exist for doing this? We can assume that it's being used for a REST API with a list of entry functions (API endpoints) that need coverage analysis, and each entry function will end with a known 'exit function' that responds to the requester in a standard way.

Here's what I've found so far:

1: Basic solution (Currently implemented)

• When writing each REST endpoint, manually create a list of all the possible outcome 'codes' [Success, FailureDueToX, FailureDueToY] for example
• After the tests have run, ensure every code in the list has been seen by the test suite for each endpoint.

Pros: Very basic and easy to use; Doesn't change performance testing times

Cons: Highly prone to error with lots of manual checking; Doesn't flag any issues if there are 5 ways to 'FailDueToX' and you only test 1 of them. Very basic definition of 'coverage'

2: Static analysis

• Parse the code into some sort of parse tree and look for all the instances of the 'exit function'
• Traverse up the tree until an API Endpoint is reached, and add that exit instance to the endpoint as an expected output (needs to keep a record of the stack trace to get there via a hash or similar)
• When tests are run, the endpoints return the stack trace hash or similar and this is compared to the expected output list.

Pros: Automatic; Catches the different branches that may result in the same output code

Cons: Generating the parse tree is non trivial; Doesn't detect dead code that never gets run; test suite needs to be kept in sync

3: Profiling

I've done this in the past on Embedded Systems with GreenHills Code Coverage Tools

• Start up a profiler like dtrace and record the stack log for each test separately
• Parse the stack log and assign the 'test' to each line of code
• Manually analyse the code-with-annotations to find gaps.

Pros: Semi Automatic; Gives more information about total coverage to a developer; Can see

Cons: Slows down the tests; Unable to do performance testing in parallel; Doesn't flag when a possible outcome is never made to happen.

What else is there, and what tools can help me with my Static Analysis and Profiling goals?

Answer 1

Combinatorial testing (different name suggestion would be appreciated)

• loosely based on idea of QuickCheck
• requires initial extraction of the endpoints (and static analysis mentioned in #2), creates list of all potential endpoints and arguments, and executes them
• validation:
  • light: app shall be stable to handle all possible inputs
  • strong: requires written specification regarding the endpoints to be able to validate against it

Pros:: Semi automatic (having proper tools)

Cons:: Specification validation is tricky. I'm not aware of existing implementation.

Potentially helpful node modules:

• claire
• quickcheck