How to control which fixture combinations are used to parametrized tests?

Question

In conftest.py, I have three fixtures.

Two of them are initialized with lists parameters.
And one of them uses the two previous fixtures like this:

@pytest.fixture(params = servers_list)
def server_fixture(request):
    server = request.param[0]
    print(server)
    yield server

@pytest.fixture(params = users_list)
def user_fixture(request):
    user = request.param[0]
    print(user)
    yield user

@pytest.fixture()
def login_fixture(server_fixture, user_fixture):
    #Some code

The lists data come from outsourcing lists and they look like this:

server_list = [path1, path2]
users_list =  [user1, user2]

I have one test in my test file (test_opertaion.py) which uses login_fixture like this:

def test_my_permission(login_fixture):
    #some code

If I run pytest, it will collect 4 tests, which cover all the possible scenarios according to my parameters:

test_my_permission[path1, user1]
test_my_permission[path1, user2]
test_my_permission[path2, user1]
test_my_permission[path2, user2]

I would like to control the scenarios to not receive all the options.

For example, I know that user2 doesn't exist on path2 so I don't want my test to run on the test_my_permission[path2, user2].

How can I do so?

I decided to create a JSON file which reflects my desired scenarios as following:

{
    "['path1';'path2']": "['user1']",
    "['path1']": "['user2']"
}

After that, I decided to write some iteration code, to read my data from the JSON file, inside the conftest.py file (maybe this is not the right place) in order to receive the desired scenarios and to set my parameters as following:

 scenario 1 : "['path1';'path2']" to servers_list parameter and
              "['user1']" to users_list parameter.
              This scenario should collect two tests.
 scenario 2 : "['path1']" to servers_list parameter and
              "['user2']" to users_list parameter.
              This scenario should collect one test.

The problem is that pytest takes only the last scenario and therefore collects only one test.

Is there anyone who knows how to solve this issue?

Alternatively, is there anyone who has an idea how to control my desired scenarios?

Answer 1

I personally would have parametrized the supported combinations/scenarios of servers and users directly into the test itself. The .parametrize function's 2nd argument accepts any iterable, so you can define your path_ and user_ pairs somewhere else, for example, define and read it from a config file:

@pytest.mark.parametrize(
    "server, user",
    [
        ("path1", "user1"),
        ("path1", "user2"),
        ("path2", "user1"),
    ],
)
def test_my_permission(server, user):
    pass

tests/test_main.py::test_my_permission[path1-user1] PASSED
tests/test_main.py::test_my_permission[path1-user2] PASSED
tests/test_main.py::test_my_permission[path2-user1] PASSED

But, since you clarified in a comment that you'd want to reuse server_fixture and user_fixture and "cannot use the parameters in the test itself.", another option is to manually check the path_ and user_ combination in login_fixture and skip it if it's not a valid combination:

@pytest.fixture()
def login_fixture(server_fixture, user_fixture):
    if server_fixture == "path2" and user_fixture == "user2":
        pytest.skip(reason="path2 and user2 combination is not valid")
    pass

def test_my_permission(login_fixture):
    pass

tests/test_main.py::test_my_permission[path1-user1] PASSED
tests/test_main.py::test_my_permission[path1-user2] PASSED
tests/test_main.py::test_my_permission[path2-user1] PASSED
tests/test_main.py::test_my_permission[path2-user2] SKIPPED (path2 and user2 combination is not valid)

This isn't that elegant, and only works as long as the skipped combination check in login_fixture applies to all the tests that uses that fixture. The upside is that the test results report clearly shows that a path2 + user2 test input was considered, but the combination is not a valid test case.

Similar to what you did, you can factor out the combinations into a JSON file:

# scenarios.json

{
    "login_scenarios": [
        ["path1", "user2"],
        ["path1", "user1"],
        ["path2", "user1"]
    ]
}

Then just read that into the test:

with open("scenarios.json") as f:
    scenarios = json.load(f)
login_scenarios = scenarios["login_scenarios"]

@pytest.fixture()
def login_fixture(server_fixture, user_fixture):
    combination = [server_fixture, user_fixture]
    if combination not in login_scenarios:
        pytest.skip(reason=f"{combination} combination is not valid")
    pass

An alternative to skipping is to implement one of pytest's collection hooks that gets called to parametrize each test: pytest_generate_tests. That hook receives a MetaFunc object, which contains the fixtures requested by each test. Based on the requested fixtures, you can dynamically set the right combination of server and user.

Put this in the conftest.py:

# In tests/conftest.py

import json

with open("scenarios.json") as f:
    scenarios = json.load(f)
login_scenarios = scenarios["login_scenarios"]

def pytest_generate_tests(metafunc):
    if {"server", "user"} <= set(metafunc.fixturenames):
        metafunc.parametrize(
            "server, user",
            login_scenarios,
        )

Then define the test and fixtures as:

# In tests/test_main.py

@pytest.fixture()
def login_fixture(server, user):
    pass

def test_my_permission(login_fixture):
    pass

That would limit the generated parametrized tests to:

tests/test_main.py::test_my_permission[path1-user1] PASSED
tests/test_main.py::test_my_permission[path1-user2] PASSED
tests/test_main.py::test_my_permission[path2-user1] PASSED

...without the path2 + user2 combination.

Again, same as before, you can define and read the test scenarios from an external file and pass it to .parametrize. Note that the hook gets called to parametrize all the tests in the target directory, so you need to check if the requesting test does require the server and user fixtures.