CODEWITHSUNDEEP
phpphpunit
Mathieu Dumoulin
Mathieu Dumoulin

Reputation: 12244

How can you assert an assertion was done

We build test helper traits in our application and they mostly apply assertion logic in a convenient way. The problem is that we can test that there are errors with these traits but we cannot assume that there are no errors and the test was passed.

I find this a little overboard but my colleage is really into testing everything even custom assertions.

I looked at the PHPUnit code to make sure there wasn't an undocumented feature regarding this such as getAssertionCount() and found in fact that there was this feature:

TestCase::getNumAssertions()

but it doesn't seem to be updated and is always at zero.

So how can you find out if an assertion was done or not?

Here's an example of the test i'm running:

/**
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::assertHasConstraintViolation
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::assertNotHasConstraintViolation
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::getViolations
 */
public function testAssertConstraintViolations()
{
    $o = new EntityWithValidators();
    $this->assertHasConstraintViolation($o, 'property', null);
    $this->assertNotHasConstraintViolation($o, 'property', 'valid value');
}

Now this code really runs "assertHasContraintViolation" but we have no way to test that the assertion ran. If the assertion failed and i caught it such as in :

/**
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::assertHasConstraintViolation
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::assertNotHasConstraintViolation
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::getViolations
 * @expectedException PHPUnit_Framework_ExpectationFailedException
 */
public function testAssertHasConstraintViolationsAssertionFailsCorrectly()
{
    $o = new EntityWithValidators();
    $this->assertHasConstraintViolation($o, 'property', 'valid value');
}

/**
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::assertHasConstraintViolation
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::assertNotHasConstraintViolation
 * @covers UgroupMedia\CommonBundle\TestHelper\ModelTestTrait::getViolations
 * @expectedException PHPUnit_Framework_ExpectationFailedException
 */
public function testAssertNotHasConstraintViolationsAssertionFailsCorrectly()
{
    $o = new EntityWithValidators();
    $this->assertNotHasConstraintViolation($o, 'property', null);
}

then this actually validates that the assertion does what it needs in a negative way. We catch the error and assertTrue(true) to state that it was ok. But we can'T assert that the assertion passed in any reasonnable way.

Upvotes: 1

Views: 368

Answers (2)

Cristik
Cristik

Reputation: 32874

So you need to test the unit-tests to make sure they're actually testing? But who checks this level 2 of testing? You'll need another level to make sure that the assertion based on the number of assertions was made. And so on...

Unit tests should not need be tested, as their implementation should be simple and linear. And missing assertions can be caught at the code review phase. If you don't trust the unit tests, you also don't trust the code, and this is a bigger problem.

Upvotes: 2

hek2mgl
hek2mgl

Reputation: 158190

I'll not go into possible implementation details, I just want to point out that it is basically a bad idea to test the number of assertions been made in a testsuite, since this value would be really hard to maintain. Let's say you have 1724 tests. You would need to track this value across every change in the test suite. This does not make sense imo.

Instead I would use $this->markTestSkipped(), $this->markTestIncomplete() and friends to make sure that everything runs as expected.

Upvotes: 2

Related Questions