Pytest Class level fixture runs for every method if there is an exception in the fixture

Question

I am writing some tests using pytest which needs the an environment variable to run the test function. So i created a fixture which would check if the environment variable exists or exit otherwise. Below is my code.

import os
import sys

import pytest

from ..factorial import fact


@pytest.fixture(scope='class')
def pre(request):
    print('************setting up************')

    pwd = os.environ.get('PASSWD', None)
    if pwd is not None:
        request.cls.pwd = pwd
    else:
        sys.exit('This test requires your password, '
                 'please run export PASSWD=<yourPassword>')


@pytest.mark.usefixtures('pre')
class TestFactorial:
    def test_postive(self):
        assert fact(5) == 120

    def test_false(self):
        assert fact(6) == 720

When the environment variable is set I get the expected output as following and the fixture actually runs once per class.

collected 2 items                                                                                                                                                                                                  

test_factorial.py ************setting up************
..

============================================================================================= 2 passed in 0.01 seconds ===

Now when the environment variable is not set, the whole fixture is run twice. Below is the output

collected 2 items                                                                                                                                                                                                  

test_factorial.py ************setting up************
E************setting up************
E

====================================================================================================== ERRORS ======================================================================================================
___________________________________________________________________________________ ERROR at setup of TestFactorial.test_postive ___________________________________________________________________________________

request = <SubRequest 'pre' for <Function test_postive>>

    @pytest.fixture(scope='class')
    def pre(request):
        print('************setting up************')

        pwd = os.environ.get('PASSWD', None)
        if pwd is not None:
            request.cls.pwd = pwd
        else:
>           sys.exit('This test requires your password, '
                     'please run export PASSWD=<yourPassword>')
E           SystemExit: This test requires your password, please run export PASSWD=<yourPassword>

test_factorial.py:17: SystemExit
____________________________________________________________________________________ ERROR at setup of TestFactorial.test_false ____________________________________________________________________________________

request = <SubRequest 'pre' for <Function test_false>>

    @pytest.fixture(scope='class')
    def pre(request):
        print('************setting up************')

        pwd = os.environ.get('PASSWD', None)
        if pwd is not None:
            request.cls.pwd = pwd
        else:
>           sys.exit('This test requires your password, '
                     'please run export PASSWD=<yourPassword>')
E           SystemExit: This test requires your password, please run export PASSWD=<yourPassword>

test_factorial.py:17: SystemExit

I was expecting that exception would be raised and program would exit and it would not even reach the TestFactorial class. But this shows that fixture actually ran twice once each test method inside the class.

Is this the expected behavior or am i missing something in understanding how fixtures actually work ?

If this is indeed the expected then how can i achieve this behavior that if the environment variable is not set tests should not run ?

Edit 1

I made some more changes into my code to actually see if the fixture runs for each method if there is an exception in the fixture and it does. Below is updated code and results

import os
import sys
from random import randint   # edit

import pytest

from ..factorial import fact


@pytest.fixture(scope='class')
def pre(request):
    print('************setting up************')

    pwd = os.environ.get('PASSWD', None)
    if pwd is not None:
        print(randint(1, 100))    # edit
        request.cls.pwd = randint(1, 100)
    else:
        print(randint(1, 100))    # edit
        sys.exit('This test requires your password, '
                 'please run export PASSWD=<yourPassword>')

@pytest.mark.usefixtures('pre')
class TestFactorial:
    def test_postive(self):
        assert fact(5) == 120

    def test_false(self):
        assert fact(6) == 720

Output when the environment variable is set

pytest -sq test_factorial.py
************setting up************
67
..
2 passed in 0.01 seconds

So it ran just once for the class.

Output when environment variable is not set

pytest -sq test_factorial.py
************setting up************
69
E************setting up************
82
E
====================================================================================================== ERRORS ======================================================================================================
___________________________________________________________________________________ ERROR at setup of TestFactorial.test_postive ___________________________________________________________________________________

request = <SubRequest 'pre' for <Function test_postive>>

    @pytest.fixture(scope='class')
    def pre(request):
        print('************setting up************')

        pwd = os.environ.get('PASSWD', None)
        if pwd is not None:
            print(randint(1, 100))
            request.cls.pwd = randint(1, 100)
        else:
            print(randint(1, 100))
>           sys.exit('This test requires your password, '
                     'please run export PASSWD=<yourPassword>')
E           SystemExit: This test requires your password, please run export PASSWD=<yourPassword>

test_factorial.py:20: SystemExit
____________________________________________________________________________________ ERROR at setup of TestFactorial.test_false ____________________________________________________________________________________

request = <SubRequest 'pre' for <Function test_false>>

    @pytest.fixture(scope='class')
    def pre(request):
        print('************setting up************')

        pwd = os.environ.get('PASSWD', None)
        if pwd is not None:
            print(randint(1, 100))
            request.cls.pwd = randint(1, 100)
        else:
            print(randint(1, 100))
>           sys.exit('This test requires your password, '
                     'please run export PASSWD=<yourPassword>')
E           SystemExit: This test requires your password, please run export PASSWD=<yourPassword>

test_factorial.py:20: SystemExit
2 error in 0.03 seconds

Note the random numbers printed in each case are different so this shows that fixture actually fired twice, once for each method in the class.

This is very weird behavior. Any clues ?

Answer 1

I was able to achieve the desired result by using pytest.skip inside the fixture. This way if the fixture throws some exception the tests using the fixtures would be skipped.

Answer 2

pytest catches the SystemExit exception, which is reasonable when you think about it: if you're testing a function that happens to call sys.exit, you wouldn't want this to shut down your whole test suite.

Use pytest.exit instead of sys.exit, which signals to pytest that you want to really shut down the test suite.