How to prevent test execution if fixture setup fails?

Question

I want to test a DB application with pytest, but I want to run my tests only if the initial connection setup in my fixture is successful. Otherwise, I simply want the test runner to pass successfully. I came up with the following code:

import logging

import MySQLdb
import pytest


@pytest.fixture('module')
def setup_db():
    try:
        conn = MySQLdb.connect("127.0.0.1", 'testuser', 'testpassword', 'testdb')
        yield conn
        conn.close()
    except Exception as e:
        logging.exception("Failed to setup test database")
        yield


def test_db(setup_db):
    if not setup_db:
        assert True   # some dummy assert to mark test as True
    else:
        assert 4 == 1 + 3

As you can see this is hacky, cumbersome, and requires all of my tests to have this boilerplate condition to check if setup_db fixture actually yielded something or not.

What I would ideally want is to maybe return None or raise some exception which pytest framework can catch and stop running the test suit. I tried returning from the fixture but it didn't work.

Any ideas?

Answer 1

For cases where your tests and fixtures have external dependencies (ex. a working database) and it's not really possible to run them successfully on a specific environment (ex. on a CI/CD service on the cloud), then it's better to simply not run those tests in the first place. Or in reverse, only run those tests that are actually runnable on that environment.

This can be done using custom pytest markers.

As a sample setup, I have a tests folder with 2 sets of tests: 1 that requires a working DB and a working setup_db fixture (test_db.py), and 1 set of tests that have no external dependencies (test_utils.py).

$ tree tests
tests
├── pytest.ini
├── test_db.py
└── test_utils.py

The pytest.ini is where you register your markers:

[pytest]
markers =
    uses_db: Tests for DB-related functions (requires a working DB)
    utils: Tests for utility functions
    cicd: Tests to be included in a CI/CD pipeline

The markers can be set on function-level or on a class/module-level. For this case, it's better to group all DB-related tests into their own files and to mark the entire file:

test_db.py

import pytest

pytestmark = [pytest.mark.uses_db]

# Define all test functions here...

test_utils.py

import pytest

pytestmark = [pytest.mark.utils, pytest.mark.cicd]

# Define all test functions here...

Here each group of tests is marked, and the test_utils.py is additionally marked with cicd indicating it's compatible with running in a CI/CD pipeline (you can name your markers any way you like, this is just an example I personally use).

Option 1

Tell pytest to run all tests except for tests marked with uses_db

$ pytest tests -v -m "not uses_db"
=========================== test session starts ===========================
...
collected 5 items / 3 deselected / 2 selected                                                                                                                                                                     

tests/test_utils.py::test_utils_1 PASSED                             [ 50%]
tests/test_utils.py::test_utils_2 PASSED                             [100%]                                                                                                                                                           

Here pytest finds all the tests not marked with uses_db and ignores them ("deselected"). The test codes and any related fixture codes are not executed. You don't have to worry about catching and handling DB exceptions. None of those codes are run in the first place.

Option 2

Tell pytest to run only tests marked with cicd

$ pytest tests -v -m "cicd"
=========================== test session starts ===========================
...
collected 5 items / 3 deselected / 2 selected                                                                                                                                                                     

tests/test_utils.py::test_utils_1 PASSED                             [ 50%]
tests/test_utils.py::test_utils_2 PASSED                             [100%]                                                                                                                                                           

The result should be the same in this case. Here, pytest find all the tests marked with cicd and ignores all the others ("deselected"). Same as in Option 1, none of the deselected tests and fixtures are run, so again you don't have to worry about catching and handling DB exceptions.

---

So, for the case where:

The issue is when I push this code, the ci/cd pipelines runs the test suit and if those machines do not have the testdb setup, the whole test suit will fail and ci/cd pipeline will abort and I don't want.

You either use Option 1 or Option 2 to select which tests to run.

This is a bit different than using .skip() (like in this answer) or ignoring fixture-related exceptions raised during the test execution because those solutions still evaluate and run the fixture and test codes. Having to skip/ignore a test because of Exception can be misleading. This answer avoids running them entirely.

It is also clearer because your run command explicitly states which sets of tests should be or should not be run, rather than relying on your test codes to handle possible Exceptions.

In addition, using markers allows using pytest --markers to give you a list of which tests can or cannot be run:

$ pytest tests --markers 
@pytest.mark.uses_db: Tests for DB-related functions (requires a working DB)

@pytest.mark.utils: Tests for utility functions

@pytest.mark.cicd: Tests to be included in a CI/CD pipeline

...other built-in markers...

Answer 2

I recommend aborting or failing the entire test run if setting up the DB fixture fails, instead of just skipping the affected tests (as in this answer) or much worse, letting the tests pass successfully (as you mentioned at the start of your question: "I simply want the test runner to pass successfully.").

A failing DB fixture is a sign that something is wrong with your test environment, so there should be no point running any of the other tests until the problems with your environment are resolved. Even if the other tests pass, it can give you false positive results or a false sense of confidence if something is wrong with your test setup. You'd want the run to catastrophically fail with a clear error state that says "Hey, your test setup is broken".

There are 2 ways to abort the entire test run:

1. Call pytest.exit

   @pytest.fixture(scope="module")
   def setup_db():
       try:
           conn = DB.connect("127.0.0.1", "testuser", "testpassword", "testdb")
           yield conn
           conn.close()
       except Exception as e:
           pytest.exit(f"Failed to setup test database: {e}", returncode=2) 
   

   It prints out:

   =========================== test session starts ============================
   ...
   collected 3 items                                                          
   
   test.py::test_db_1  
   
   ========================== no tests ran in 0.30s ===========================
   !!! _pytest.outcomes.Exit: Failed to setup test database: Timeout Error !!!!
   

   As shown, there should be 3 tests but it stopped after the 1st one failed to load the fixture. One nice thing about .exit() is the returncode parameter, which you can set to some error code value (typically some non-zero integer value). This works nicely with automated test runners, merge/pull request hooks, and CI/CD pipelines, because those typically check for non-zero exit codes.

2. Pass the -x or --exitfirst option

   Code:

   @pytest.fixture(scope="module")
   def setup_db():
       # Don't catch the error
       conn = DB.connect("127.0.0.1", "testuser", "testpassword", "testdb")
       yield conn
       conn.close()
   

   Run:

   $ pytest test.py -v -x
   =========================== test session starts ===========================
   ...
   collected 3 items                                                                                          
   
   test.py::test_db_1 ERROR                                         [ 33%]
   
   ================================= ERRORS ==================================
   _______________________ ERROR at setup of test_db_1 _______________________
   
       @pytest.fixture(scope="module")
       def setup_db():
   >       conn = DB.connect("127.0.0.1", "testuser", "testpassword", "testdb")
   
   test.py:30: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _  
   
   ip = '127.0.0.1', username = 'testuser', password = 'testpassword', 
   db_name = 'testdb'
   
       @staticmethod
       def connect(ip: str, username: str, password: str, db_name: str):
           ...
   >       raise Exception("Cannot connect to DB: Timeout Error")
   E       Exception: Cannot connect to DB: Timeout Error
   
   test.py:16: Exception
   ======================= short test summary info ========================
   ERROR test.py::test_db_1 - Exception: Cannot connect to DB: Timeout Error
   !!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!
   =========================== 1 error in 0.05s ===========================
   

   Here, you don't catch the error in the fixture. Just let the Exception be raised and let it be unhandled, so that pytest will see that as a failed test, and then -x/--exitfirst aborts everything else on the 1st failed test.

   Similar to pytext.exit(), the return/exit code will also be non-zero, so this works as well with automated test runners, merge/pull request hooks, and CI/CD pipelines.

Answer 3

Thanks to @Mrbean Bremen suggestion I was able to skip tests from the fixture itself.

@pytest.fixture('module')
def setup_db():
    try:
        conn = MySQLdb.connect("127.0.0.1", 'testuse', 'testpassword', 'testdb')
        yield conn
        conn.close()
    except Exception as e:
        logging.exception("Failed to setup test database")
        pytest.skip("****** DB Setup failed, skipping test suit *******", allow_module_level=True)