CODEWITHSUNDEEP
pythonpython-3.xpytest
user4093955
user4093955

Reputation:

Mixing parametrized tests and marks

Ok, I'm struggling with something that is literally blowing my mind.

Although my actual code is different, this basically nails down the issue. Assume this sample code:

import pytest

@pytest.mark.parametrize('type',(
    pytest.param('stability', marks=pytest.mark.stability),
    pytest.param('integration', marks=pytest.mark.integration),
))
@pytest.mark.integration
@pytest.mark.stability
def test_meh(type):
    assert type == 'integration'

And the following is the output of running that test. 

$pytest -m integration test_meeh.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.6.4, pytest-3.1.2, py-1.5.2, pluggy-0.4.0
rootdir: /Users/yzT/Desktop, inifile:
collected 2 items

test_meeh.py F

============================================================== FAILURES ===============================================================
_________________________________________________________ test_meh[stability] _________________________________________________________

type = 'stability'

    @pytest.mark.parametrize('type',(
        pytest.param('stability', marks=pytest.mark.stability),
        pytest.param('integration', marks=pytest.mark.integration),
    ))
    @pytest.mark.integration
    @pytest.mark.stability
    def test_meh(type):
>       assert type == 'integration'
E       AssertionError: assert 'stability' == 'integration'
E         - stability
E         + integration

test_meeh.py:10: AssertionError
========================================================= 1 tests deselected ==========================================================
=============================================== 1 failed, 1 deselected in 0.07 seconds ================================================


$ pytest -m stability test_meeh.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.6.4, pytest-3.1.2, py-1.5.2, pluggy-0.4.0
rootdir: /Users/yzT/Desktop, inifile:
collected 2 items

test_meeh.py .

========================================================= 1 tests deselected ==========================================================
=============================================== 1 passed, 1 deselected in 0.02 seconds ================================================

What is going on here? Why when I use -m integration it uses stability and when I use -m stability it uses integration ?

Upvotes: 3

Views: 265

Answers (1)

Samuel Dion-Girardeau
Samuel Dion-Girardeau

Reputation: 3170

Looks like there are two issues with the code sample that you provided.

First, you shouldn't mark both individual parametrized tests and the test function. As soon as the -m option you provided matches the mark decorator on the function, the test will be selected.

Here's a minimal comparison. With both the marks on the function and individual parametrized tests:

# parametrized_tests.py
import pytest
@pytest.mark.parametrize('smiley', [
    pytest.param(':)', marks=[pytest.mark.happy]),
    pytest.param(':(', marks=[pytest.mark.unhappy]),
])
@pytest.mark.happy
@pytest.mark.unhappy
def test_meh(smiley):
    assert smiley == ':)'

You will have two tests collected and selected:

$ pytest -m happy --collect-only parametrized_tests.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items
<Module 'params-individual-tests.py'>
  <Function 'test_smiley[:)]'>
  <Function 'test_smiley[:(]'>

====================================================================================== no tests ran in 0.01 seconds ======================================================================================

But if only the parametrized tests are marked:

# parametrized_tests.py
import pytest

@pytest.mark.parametrize('smiley', [
    pytest.param(':)', marks=[pytest.mark.happy]),
    pytest.param(':(', marks=[pytest.mark.unhappy]),
])
def test_smiley(smiley):
    assert smiley == ':)'

You will get two tests collected, but only one selected, as you expect:

$ pytest -m happy --collect-only parametrized_tests.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items / 1 deselected
<Module 'params-individual-tests.py'>
  <Function 'test_smiley[:)]'>

====================================================================================== 1 deselected in 0.01 seconds ======================================================================================

Second, there seems to be a puzzling bug (or undocumented "feature") in pytest.param, where using the exact name of the mark as the parameter value makes the test not selected.

From your (slightly modified) code:

# mark_name.py
import pytest

@pytest.mark.parametrize('type_', [
    pytest.param('integration', marks=[pytest.mark.integration]),
    pytest.param('stability', marks=[pytest.mark.stability]),
])
def test_meh(type_):
    assert type_ == 'integration'

If I try to run the integration tests only, it won't select any:

$ pytest -m integration mark_name.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items / 2 deselected

But simply modifying the value (in this case I only made it uppercase) makes everything work as expected:

# mark_name.py
import pytest

@pytest.mark.parametrize('type_', [
    pytest.param('INTEGRATION', marks=[pytest.mark.integration]),
    pytest.param('STABILITY', marks=[pytest.mark.stability]),
])
def test_meh(type_):
    assert type_ == 'INTEGRATION'

And now I can select the tests properly:

$ pytest -m integration mark_name.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items / 1 deselected

mark_name.py .                                                                                                                                                                                     [100%]

================================================================================= 1 passed, 1 deselected in 0.01 seconds =================================================================================
$ pytest -m stability mark_name.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: [redacted], inifile:
collected 2 items / 1 deselected

mark_name.py F                                                                                                                                                                                     [100%]

================================================================================================ FAILURES ================================================================================================
__________________________________________________________________________________________ test_meh[STABILITY] ___________________________________________________________________________________________

type_ = 'STABILITY'

    @pytest.mark.parametrize('type_', [
        pytest.param('INTEGRATION', marks=[pytest.mark.integration]),
        pytest.param('STABILITY', marks=[pytest.mark.stability]),
    ])
    def test_meh(type_):
>       assert type_ == 'INTEGRATION'
E       AssertionError: assert 'STABILITY' == 'INTEGRATION'
E         - STABILITY
E         + INTEGRATION

mark_name.py:9: AssertionError
================================================================================= 1 failed, 1 deselected in 0.08 seconds =================================================================================

I suspect that's related to the string value being used as the ID of the test, but would suggest opening a GitHub issue if that's something you don't want to work around.

Also, totally unrelated, but it's better not to define the argument as type, because you are shadowing the builtin function type. PEP-8 suggests using a trailing underscore to prevent a name clash.

Upvotes: 2

Related Questions