CODEWITHSUNDEEP
pythonpytest
sorin
sorin

Reputation: 170410

How to make pytest return failure if a minimum number of tests is not run?

As tests can be enabled or disabled automatically based on existing environment there is always the risk that something in CI is changed that makes pytest skip tests when they were not supposed to.

If I could define a minimum number of tests, or maybe even a fixed number of expected tests in an environment variable I could prevent accidents where CI passes just because tests were skipped.

Most of the time pytest is called by tox, which is also used in development environments. This means that the expected number of tests must be defined in CI config (like .travis.yml), because this is the only place which is specific to CI.

Upvotes: 1

Views: 1087

Answers (1)

hoefling
hoefling

Reputation: 66231

You can add a custom hook impl and count the amount of passed tests. After the test run finishes, check the counter and exit with a custom exit code if the amount of tests passed was too low. Example: add the code to your conftest.py:

import pytest


def pytest_addoption(parser):
    parser.addoption('--minpass', type=int, default=0, help='minimum amount of tests to pass')


def pytest_sessionstart(session):
    session.count_passed = 0


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    result = outcome.get_result()

    if result.when == 'call' and result.passed:
        item.session.count_passed += 1


def pytest_sessionfinish(session, exitstatus):
    min_passed = session.config.getoption('minpass')
    if session.count_passed < min_passed:
        session.exitstatus = 127
        reporter = session.config.pluginmanager.get_plugin('terminalreporter')
        reporter.section('Session errors', sep='-', red=True, bold=True)
        reporter.line(f'Not enough successful tests - expected at least {min_passed} to pass, passed {session.count_passed}')

This adds a new command line option --minpass that expects a minimal amount of tests required for the run to succeed; the default value is zero so it shouldn't affect any existing suites if not provided. session.exitstatus = 127 sets the custom exit code; change it to whatever code you need. The pytest_sessionfinish hook uses the terminal reporter to display the error in a custom section; change the output to whatever format you need.

A sample execution with at least one test to pass yields:

$ pytest --minpass 1
======================================= test session starts ========================================
platform linux -- Python 3.6.9, pytest-5.3.1, py-1.8.0, pluggy-0.13.0
rootdir: /home/hoefling/projects/private/stackoverflow/so-59116898
plugins: mock-1.11.0, xdist-1.29.0, asyncio-0.10.0, forked-1.0.2, cov-2.8.1, testinfra-3.2.0
collected 1 item                                                                                   

test_spam.py s                                                                               [100%]
------------------------------------------ Session errors ------------------------------------------
Not enough successful tests - expected at least 1 to pass, passed 0


======================================== 1 skipped in 0.01s ========================================
$ echo $?
127

If you want to support the environment variables, adapt the pytest_sessionfinish hook impl with e.g.

def pytest_sessionfinish(session, exitstatus):
    min_passed = session.config.getoption('minpass') or int(os.environ.get('PYTEST_MINPASS', '0'))
    ...

and set the PYTEST_MINPASS environment variable to turn on the check.

Upvotes: 2

Related Questions