Is it possible to execute the same test in parallel in pytest?

Question

I understand that you can parameterize a test to repeat the test with different set of parameters. Also I know that different tests in a test file can be run in parallel using the -n, but I would like to execute the same set of tests in parallel. Is it possible in pytest?

For example:

import pytest

@pytest.fixture()
def user_number(worker_id):
    return "user number : %s" %worker_id

def test_add(user_number):
    print("Adding 1+1 and returning the result and user number: {}".format(user_number))
    return 1+1

def test_subtract():
    print("subtracting 2-1 and returning the result and user number: {}".format(user_number))
    return 2-1

If I run the following command: py.test -n 3 -s -v parallel_users.py

In the result, test_add() and test_subtract() are run in parallel as shown below:

[gw1] PASSED parallel_users.py::test_subtract 
[gw0] PASSED parallel_users.py::test_add

How can I get test_add() and test_subtract() run twice, something like below:

[gw1] PASSED parallel_users.py::test_add, test_subtract 
[gw0] PASSED parallel_users.py::test_add, test_subtract 

Answer 1

Update:

I guess the closest to what the OP is looking for is using the each distscope. Using it will execute the test selection n times, once per each process:

$ pytest -n3 -v --dist=each
...
gw0 [2] / gw1 [2] / gw2 [2]
scheduling tests via EachScheduling

test_main.py::test_add 
[gw1] [ 50%] PASSED test_main.py::test_add 
[gw0] [ 50%] PASSED test_main.py::test_add 
test_main.py::test_subtract 
[gw2] [ 50%] PASSED test_main.py::test_add 
test_main.py::test_subtract 
[gw2] [100%] PASSED test_main.py::test_subtract 
[gw0] [100%] PASSED test_main.py::test_subtract 
[gw1] [100%] PASSED test_main.py::test_subtract 

Both test_add and test_subtract were executed once in each worker gw0, gw1 and gw2, summing up to three executions for each test.

Old answer

To repeat test execution, add a hook in your conftest.py:

def pytest_collection_modifyitems(items):
    numrepeats = 2
    items.extend(items * (numrepeats - 1))

This will copy each of the tests collected for the execution numrepeats times. Example run:

$ pytest test_spam.py -v -n3
============================= test session starts ==============================
platform darwin -- Python 3.6.4, pytest-3.4.2, py-1.5.3, pluggy-0.6.0 -- /Users/hoefling/.virtualenvs/stackoverflow/bin/python3.6
cachedir: .pytest_cache
rootdir: /private/tmp, inifile:
plugins: xdist-1.22.2, forked-0.2, dependency-0.3.2, cov-2.5.1
[gw0] darwin Python 3.6.4 cwd: /private/tmp
[gw1] darwin Python 3.6.4 cwd: /private/tmp
[gw2] darwin Python 3.6.4 cwd: /private/tmp
[gw0] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw1] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw2] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
gw0 [4] / gw1 [4] / gw2 [4]
scheduling tests via LoadScheduling

test_spam.py::test_add
test_spam.py::test_subtract
test_spam.py::test_add
[gw0] [ 25%] PASSED test_spam.py::test_add
[gw1] [ 50%] PASSED test_spam.py::test_subtract
[gw2] [ 50%] PASSED test_spam.py::test_add
test_spam.py::test_subtract
[gw0] [ 50%] PASSED test_spam.py::test_subtract

=========================== 4 passed in 0.63 seconds ===========================

If you want to make it configurable, add a custom cli argument:

import pytest

def pytest_addoption(parser):
    parser.addoption('--numrepeats', action='store', type=int, default=1)

def pytest_collection_modifyitems(items):
    numrepeats = pytest.config.getoption('--numrepeats')
    items.extend(items * (numrepeats - 1))

Now you can call your tests with --numrepeats, for example pytest --numrepeats 5.

---

As for batching tests per process (second part of your question), pytest-xdist doesn't support it yet, see this issue and all the stuff linked to it. Recently, some rudimentary support like executing tests in a single module or class in a separate process was added:

--dist=distmode       set mode for distributing tests to exec environments.
                      each: send each test to all available environments.
                      load: load balance by sending any pending test to any
                      available environment. loadscope: load balance by
                      sending pending groups of tests in the same scope to
                      any available environment. loadfile: load balance by
                      sending test grouped by file to any available
                      environment. (default) no: run tests inprocess, don't
                      distribute.

However, if you want to loadbalance the tests based on some custom condition, there's no way other than writing your own scheduler impl.

Answer 2

One simplest way to achieve this is to trick py.test to assume there are multiple combination of tests using pytest.mark.parametrization. Here I am just using dummy parameter count to trick py.test there are multiple variants of the test. This is simplest approach, If you have more conditions for your tests , you can use pytest_generate_tests(metafunc): to fine tune your needs.

import pytest

@pytest.fixture()
def user_number(worker_id):
    return "user number : %s" %worker_id

@pytest.mark.parametrize("count", [1, 2,3])
def test_add(user_number, count):
    print("Adding 1+1 and returning the result and user number: {}".format(user_number))
    return 1+1

@pytest.mark.parametrize("count", [1, 2 ,3])
def test_subtract(count):
    print("subtracting 2-1 and returning the result and user number: {}".format(user_number))
    return 2-1

$ py.test test.py -n 2 -s -vv
======================================== test session starts ========================================
platform darwin -- Python 2.7.13, pytest-2.9.2, py-1.5.3, pluggy-0.3.1 -- bin/python2.7
cachedir: .cache
rootdir: , inifile:
plugins: xdist-1.17.1, repeat-0.4.1, cov-1.8.1
[gw0] darwin Python 2.7.13 cwd: 
[gw1] darwin Python 2.7.13 cwd: 
[gw1] Python 2.7.13 (default, Dec 17 2016, 23:03:43)  -- [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
[gw0] Python 2.7.13 (default, Dec 17 2016, 23:03:43)  -- [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
gw0 [6] / gw1 [6]
scheduling tests via LoadScheduling

test.py::test_add[1]
test.py::test_add[2]
[gw1] PASSED test.py::test_add[2]
[gw0] PASSED test.py::test_add[1]
test.py::test_subtract[1]
[gw1] PASSED test.py::test_subtract[1]
test.py::test_add[3]
[gw0] PASSED test.py::test_add[3]
test.py::test_subtract[3]
test.py::test_subtract[2]
[gw0] PASSED test.py::test_subtract[3]
[gw1] PASSED test.py::test_subtract[2]