Mark inputs in pytest parametrize

Question

I have a test that I would like to run as part of two different test suites with different parameter inputs based on the test suite. The test suites are identified with pytest markers.

Is there a way to mark parametrize entries so that they are only run during that specific test suite?

Here is what I'd like to do:

@pytest.mark.suiteA # include the test in Suite A
@pytest.mark.suiteB # include the test in Suite B
@pytest.mark.parametrize("inputParameter", [
                              (10),                    # use this input for Suites A and B
                              pytest.mark.suiteB(12)]) # only use this input for Suite B
def test_printInputParameter(inputParameter):
    print inputParameter

Running code like this doesn't produce the results I want-- both inputs are used for both suites.

I have seen that pytest will allow use of xfail or skip within parametrize (see "Skip/xfail with parametrize" on http://pytest.org/latest/skipping.html) If there is a way to write a conditional statement that will evaluate as true only when running Suite B, that would also accomplish what I need.

Thanks in advance for the help.

Answer 1

You are on the right track. Part of the problem is that you've marked the test function with both suiteA and suiteB, so the function is considered part of both.

The parameter marker style you are using probably worked in '14 (and still works at late as pytest 3), but it doesn't work in the latest pytest version (5.x).

These examples use the latest style, which also works at least as far back as pytest 3.x.

From the pytest docs:

When using parametrize, applying a mark will make it apply to each individual test. However it is also possible to apply a marker to an individual test instance:

import pytest


@pytest.mark.foo
@pytest.mark.parametrize(
    ("n", "expected"), [(1, 2), pytest.param(1, 3, marks=pytest.mark.bar), (2, 3)]
)
def test_increment(n, expected):
    assert n + 1 == expected

In the above example, using mark foo will run that test will all parameters (including bar). Using mark bar will run that test with only the 1 marked parameter.

So, for your example, you can do (forgive me, I updated all names, except the markers, to PEP8 standard):

@pytest.mark.parametrize("input_parameter", [
        # use this input for Suite A and Suite B
        pytest.param(10, marks=[pytest.mark.suiteA, pytest.mark.suiteB]), 
        # use this input only for Suite B
        pytest.param(12, marks=pytest.mark.suiteB),
        # this input will run when no markers are specified
        (13),
]) 
def test_print_input_parameter(input_parameter):
    print(input_parameter)

You must get rid of the two mark decorators above the function. You will only need this parameter decorator.

Based on the comments, input 10 is marked to ensure it will only run with suiteA or suiteB. It will execute if either of these, or both, are called for.

Input 12 is tied to a single mark, suiteB. It will only execute if suiteB is called for.

I also added input value 13 as an example for the default unmarked test runs. As normal, this will not execute for suiteA or suiteB (or suiteC or any other marker filter), but will run if no markers are specified (as will the rest).

Alternatively, you could do:

@pytest.mark.suiteB # this function is a part of suiteB
@pytest.mark.parametrize("input_parameter", [
        # use this input for Suite A and Suite B
        pytest.param(10, marks=pytest.mark.suiteA), 
        # use this input only for Suite B
        (12),
        # this input is also part of Suite B thanks to the function decorator, as well as suiteC and suiteD
        pytest.param(13, marks=[pytest.mark.suiteC, pytest.mark.suiteD]),
]) 
def test_print_input_parameter(input_parameter):
    print(input_parameter)

With your original two parameters, you really had a full suiteB test with one parameter only for suiteA.

In this case, the function decorator runs the whole test under suiteB. If suiteA is specified, only 10 will be executed.

Because the function decorator is used, my made up parameter 13, like all parameters for this function, is also part of suiteB. I can add it to as many other markers as I want, but the function decorator ensures this test will run with all parameters under suiteB.

This alternative would work given your example, but if you have any parameters that do not overlap (e.g. 13 is not run under suiteB), you'll have to specify each of them individually like the middle example.

Answer 2

you have with @pytest.mark.suiteX made all parametrized tests have that mark, so in effect, you have marked all tests, instead apply marks only inside the list of parameters:

import pytest

mark = pytest.mark

@mark.parametrize("inputParameter", [
    mark.suiteA(10),
    mark.suiteB(12),
])
def test_printInputParameter(inputParameter):
    print inputParameter

then on cli select the test you want to filter on using -m (mark):

bash-4.4$ pytest -m suiteA test_in.py -sv
test_in.py::test_printInputParameter[10] 10
PASSED

bash-4.4$ pytest -m suiteB test_in.py -sv
test_in.py::test_printInputParameter[12] 12
PASSED

Answer 3

It seems like you can do this with skipif mark as described in http://pytest.org/latest/skipping.html#skip-xfail-with-parametrize (which you refer to). All you need to do is have a method of knowing if your code is running in suiteA or suiteB which you probably already have since you have the suiteA and SuiteB marks working. So for the example let's set an (ugly) helper attribute on the sys module similar as you would do to detect if your code is running under py.test:

# E.g. in conftest.py; in real life the value probably comes from a
# command line option added by pytest_addoption()
def pytest_configure(config):
    sys._my_test_suite = 'A'  # or 'B'

# The actual test can now use this in skipif
@pytest.mark.suiteA # include the test in Suite A
@pytest.mark.suiteB # include the test in Suite B
@pytest.mark.parametrize(
    "inputParameter",
    [(10), pytest.mark.skipif(sys._my_test_suite == 'A', reason='suite A only')(12)])])
def test_printInputParameter(inputParameter):
    print inputParameter

Yes, setting an attribute on the sys module is an ugly hack, but it's an easy solution for this case.