How to control test case execution order in pytest?

Question

I have two files in a directory. In one of the files there is a long-running test case that generates some output. In the other file there is a test case that reads that output.

How can I ensure the proper execution order of the two test cases? Is there any alternative other than puting the test cases in the same file in the proper order?

Answer 1

To me the simpliest way to fix tests execution order is to use them as fixtures, which are executed ordered by design.

@pytest.fixture()
def test_a():
    print("running test a first")

def test_b(test_a):
    print("running test b after test a")

Mark test dependency as fixture and pass it as an argument to dependent one.

Answer 2

Function import order also determines execution order. In the following use case, where reusable tests are imported into a module, the import order takes precedence over invocation order.

from livy.reusable_livy_tests import (
    test_delete_livy_session,
    test_create_livy_session,
    test_get_livy_session,
    test_livy_session_started,
    test_computation_of_a_statement,
)

test_create_livy_session
test_get_livy_session
test_livy_session_started
test_computation_of_a_statement
test_delete_livy_session

In the example above, test_delete_livy_session runs first, because it was imported first, despite being invoked last. I observed this today in python version 3.6.8, pytest version 7.0.0.

Answer 3

A pure-pytest solution is to use pytest.mark.parametrize, passing it a willfully ordered list of sub-functions to invoke. In the problem stated in this question, you have to import the test functions from your files (or into one of them). Either way, you'll need an __init__.py file.

Here's a minimal working example, sans imports:

tests/
- __init__.py  (empty)
- tests_that_need_ordering.py
- test_run_ordered.py

# tests_that_need_ordering.py
def C():
    assert False

def A():
    assert False

def B():
    pass

# test_run_ordered.py
import pytest

from .tests_that_need_ordering import A, B, C


@pytest.mark.parametrize("func", [C, B, A])
def test_things(func):
    func()

Notice that

• the sub-function file name does not start or end with test - otherwise it'll get picked up directly by pytest. Similarly, the tests inside have names that wouldn't be picked up if pytest looked.
• the function definition order is C A B. If they were plain test_* functions in a test_* file, pytest would have invoked them in this order.
• the deliberate parametrized order is C B A.

Output from running with verbosity -vv shows that they ran in the deliberate order C B A, and that failures are reported as expected:

tests/test_run_ordered.py::test_things[C] FAILED
tests/test_run_ordered.py::test_things[B] PASSED
tests/test_run_ordered.py::test_things[A] FAILED

---

For future reference, I did this with Python 3.10.13, pytest-7.4.2

Answer 4

@swimmer answer is good to go. Here is the polished code.

Ensure test items run first

# conftest.py
def pytest_collection_modifyitems(items):
    """test items come to first"""
    run_first = ["tests.test_b", "tests.test_c", "tests.test_a"]
    modules = {item: item.module.__name__ for item in items}
    items[:] = sorted(
        items, key=lambda x: run_first.index(modules[x]) if modules[x] in run_first else len(items)
    )

Ensure test item run last

# conftest.py
def pytest_collection_modifyitems(items):
    """test items come to last"""
    run_last = ["tests.test_b", "tests.test_c", "tests.test_a"]
    modules = {item: item.module.__name__ for item in items}
    items[:] = sorted(
        items, key=lambda x: run_last.index(modules[x]) if modules[x] in run_last else -1
)

Answer 5

As indicated by @Frank T in the accepted answer, the pytest_collection_modifyitems hook hook allows to modify the order of collected tests (items) in place. This approach has the advantage of not requiring any third party library.

A full example on how to enforce test case execution order by test class is already available in this answer.

In this case, though, it seems like you would like to enforce execution order by test module (i.e., the .py file where the test lives). The following adaptation would allow you to do so:

# conftest.py
def pytest_collection_modifyitems(items):
    """Modifies test items in place to ensure test modules run in a given order."""
    MODULE_ORDER = ["tests.test_b", "tests.test_c", "tests.test_a"]
    module_mapping = {item: item.module.__name__ for item in items}

    sorted_items = items.copy()
    # Iteratively move tests of each module to the end of the test queue
    for module in MODULE_ORDER:
        sorted_items = [it for it in sorted_items if module_mapping[it] != module] + [
            it for it in sorted_items if module_mapping[it] == module
        ]
    items[:] = sorted_items

Placing the snippet above in conftest.py replaces the default alphabetical test execution order test_a -> test_b -> test_c with test_b -> test_c -> test_a. Modules can live in different test sub-directories, and the order of tests inside a module is left unchanged.

Answer 6

Pytest's fixtures can be used to order tests in a similar way they order the creation of fixtures. While this is unconventional, it takes advantage of knowledge you may already have of the fixture system, it does not require a separate package and is unlikely to be altered by pytest plugins.

@pytest.fixture(scope='session')
def test_A():
    pass

@pytest.mark.usefixtures('test_A')
def test_B():
    pass

The scope prevents multiple calls to test_A if there are multiple tests that depend on it.

Answer 7

It's important to keep in mind, while trying to fix pytest ordering "issue", that running tests in the same order as they are specified seems to be the default behavior of pytest.

It turns out that my tests were out of that order because of one of these packages - pytest-dependency, pytest-depends, pytest-order. Once I uninstalled them all with pip uninstall package_name, the problem was gone. Looks like they have side effects

Answer 8

In general you can configure the behavior of basically any part of pytest using its well-specified hooks.

In your case, you want the "pytest_collection_modifyitems" hook, which lets you re-order collected tests in place.

That said, it does seem like ordering your tests should be easier -- this is Python after all! So I wrote a plugin for ordering tests: "pytest-ordering". Check out the docs or install it from pypi. Right now I recommend using @pytest.mark.first and @pytest.mark.second, or one of the @pytest.mark.order# markers, but I have some ideas about more useful APIs. Suggestions welcome :)

Edit: pytest-ordering seems abandoned at the moment, you can also check out pytest-order (a fork of the original project by the author).

Edit2: In pytest-order, only one marker (order) is supported, and the mentioned examples would read @pytest.mark.order("first"), @pytest.mark.order("second"), or @pytest.mark.order(#) (with # being any number).

Answer 9

Maybe you can consider using dependency pytest plugin, where you can set the test dependencies easily.

Be careful - the comments suggest this does not work for everyone.

@pytest.mark.dependency()
def test_long():
    pass

@pytest.mark.dependency(depends=['test_long'])
def test_short():
    pass

This way test_short will only execute if test_long is success and force the execution sequence as well.

Answer 10

Make use of the '--randomly-dont-reorganize' option or '-p no:randomly' available in pytest-randomly plugin, this will just run your test in the same order as you mentioned in your module.

Module:

import pytest

def test_three():
    assert True

def test_four():
    assert True

def test_two():
    assert True

def test_one():
    assert True

Execution:

(tmp.w95BqE188N) rkalaiselvan@dev-rkalaiselvan:~/$ py.test --randomly-dont-reorganize test_dumm.py
======================================================================== test session starts ========================================================================
platform linux2 -- Python 2.7.12, pytest-3.10.1, py-1.5.4, pluggy-0.7.1 -- /tmp/tmp.w95BqE188N/bin/python2
cachedir: .pytest_cache
Using --randomly-seed=1566829391
rootdir: /home/rkalaiselvan, inifile: pytest.ini
plugins: randomly-1.2.3, timeout-1.3.1, cov-2.6.0, mock-1.10.0, ordering-0.6
collected 4 items

test_dumm.py::test_three PASSED
test_dumm.py::test_four PASSED
test_dumm.py::test_two PASSED
test_dumm.py::test_one PASSED

(tmp.w95BqE188N) rkalaiselvan@dev-rkalaiselvan:~/$ py.test -p no:randomly test_dumm.py
======================================================================== test session starts ========================================================================
platform linux2 -- Python 2.7.12, pytest-3.10.1, py-1.5.4, pluggy-0.7.1 -- /tmp/tmp.w95BqE188N/bin/python2
cachedir: .pytest_cache
Using --randomly-seed=1566829391
rootdir: /home/rkalaiselvan, inifile: pytest.ini
plugins: randomly-1.2.3, timeout-1.3.1, cov-2.6.0, mock-1.10.0, ordering-0.6
collected 4 items

test_dumm.py::test_three PASSED
test_dumm.py::test_four PASSED
test_dumm.py::test_two PASSED
test_dumm.py::test_one PASSED

Answer 11

main.py:

import functools
import pytest
from demo import test_foo,test_hi

def check_depends(depends):
    try:
        for dep in depends:
            dep()
    except Exception as e:
        return dep
    else:
        return True

def pytest_depend(depends):
    def pytest_depend_decorator(func):
        stat = check_depends(depends)
        if stat is True:
            return func
        else:
            return pytest.mark.skip(True, reason="%s[skip] --> %s[Failed]" % (func.__name__, stat.__name__))(func)
    return pytest_depend_decorator


@pytest_depend([test_foo,test_hi])
def test_bar():
    pass

@pytest_depend([test_foo,test_hi])
def test_bar2():
    pass

demo.py:

def test_hi():
    pass
def test_foo():
    assert False

---

platform linux -- Python 3.5.2, pytest-3.8.2, py-1.6.0, pluggy-0.7.1 -- /usr/bin/python3

pytest -vrsx ./plugin.py