how to execute multiple tests on multiple items with py.test

Question

I'm a novice in python and also in py.test. I'm searching a way to run multiple tests on multiple items and cannot find it. I'm sure it's quite simple when you know how to do it.

I have simplified what I'm trying to do to make it simple to understand.

If I have a Test class who defines a serie of tests like this one :

class SeriesOfTests:
    def test_greater_than_30(self, itemNo):
        assert (itemNo > 30), "not greather than 30"
    def test_lesser_than_30(self, itemNo):
        assert (itemNo < 30), "not lesser thant 30"
    def test_modulo_2(self, itemNo):
        assert (itemNo % 2) == 0, "not divisible by 2"

I want to execute this SeriesOfTest on each item obtained from a function like :

def getItemNo():
    return [0,11,33]

The result i'm trying to obtain is something like :

RESULT : 
Test "itemNo = 0"
  - test_greater_than_30 = failed
  - test_lesser_than_30 = success
  - test_modulo_2 = success

Test "itemNo = 11"
  - test_greater_than_30 = failed
  - test_lesser_than_30 = success
  - test_modulo_2 = failed

Test "itemNo = 33"
  - test_greater_than_30 = success
  - test_lesser_than_30 = failed
  - test_modulo_2 = failed

How can I do this with py.test?

Than you guys (and girls also)

André

Answer 1

Forget about the previous answer. Given that you need the tests to be grouped by value, you can use scenarios. I've just adapted the example from the docs:

import pytest

def pytest_generate_tests(metafunc):
    idlist = []
    argvalues = []
    for scenario in metafunc.cls.scenarios:
        idlist.append(scenario[0])
        items = scenario[1].items()
        argnames = [x[0] for x in items]
        argvalues.append(([x[1] for x in items]))
    metafunc.parametrize(argnames, argvalues, ids=idlist, scope="class")

scenario1 = ('itemNo = 0', {'itemNo': 0})
scenario2 = ('itemNo = 11', {'itemNo': 11})
scenario3 = ('itemNo = 33', {'itemNo': 33})

class TestSeries:
    scenarios = [scenario1, scenario2, scenario3]
    
    def test_greater_than_30(self, itemNo):
        assert (itemNo > 30), "not greather than 30"
    def test_lesser_than_30(self, itemNo):
        assert (itemNo < 30), "not lesser thant 30"
    def test_modulo_2(self, itemNo):
        assert (itemNo % 2) == 0, "not divisible by 2"

And the output is:

$ py.test -v
============ test session starts ==============================================
platform linux2 -- Python 2.7.4 -- pytest-2.4.2 -- /home/jose/.virtualenvs/pytest1/bin/python
collected 9 items 

test_first.py:23: TestSeries.test_greater_than_30[itemNo = 0] FAILED
test_first.py:25: TestSeries.test_lesser_than_30[itemNo = 0] PASSED
test_first.py:27: TestSeries.test_modulo_2[itemNo = 0] PASSED
test_first.py:23: TestSeries.test_greater_than_30[itemNo = 11] FAILED
test_first.py:25: TestSeries.test_lesser_than_30[itemNo = 11] PASSED
test_first.py:27: TestSeries.test_modulo_2[itemNo = 11] FAILED
test_first.py:23: TestSeries.test_greater_than_30[itemNo = 33] PASSED
test_first.py:25: TestSeries.test_lesser_than_30[itemNo = 33] FAILED
test_first.py:27: TestSeries.test_modulo_2[itemNo = 33] FAILED

I think that's the closest you can get.

Answer 2

Use fixture:

import pytest

@pytest.fixture(params=[0, 11, 33])
def itemNo(request):
    return request.param

def test_greater_than_30(itemNo):
    assert (itemNo > 30), "not greather than 30"
def test_lesser_than_30(itemNo):
    assert (itemNo < 30), "not lesser thant 30"
def test_modulo_2(itemNo):
    assert (itemNo % 2) == 0, "not divisible by 2"

NOTE: The name of the fixture function (itemNo) and the name of the parameter of test functions should be same.

See Demo run.

---

UPDATE

import pytest

class FunctionWrapper(str):
    def __init__(self, f):
        self.f = f
    def __call__(self, *args, **kwargs):
        return self.f(*args, **kwargs)
    def __str__(self):
        return self.f.__name__

def greater_than_30(itemNo):
    assert (itemNo > 30), "not greater than 30"
def lesser_than_30(itemNo):
    assert (itemNo < 30), "not lesser thant 30"
def modulo_2(itemNo):
    assert (itemNo % 2) == 0, "not divisible by 2"

@pytest.fixture(params=[0, 11, 33])
def itemNo(request):
    return request.param

@pytest.fixture(params=map(FunctionWrapper, [
    greater_than_30, lesser_than_30, modulo_2
]))
def assertion_func(request):
    return request.param

def test_item_no(itemNo, assertion_func):
    assertion_func(itemNo)

Use -v --tb=no option.

For example:

============================= test session starts ==============================
platform linux2 -- Python 2.7.5 -- pytest-2.3.5 -- /usr/bin/python
collected 9 items

test_sample.py:26: test_item_no[0-greater_than_30] FAILED
test_sample.py:26: test_item_no[0-lesser_than_30] PASSED
test_sample.py:26: test_item_no[0-modulo_2] PASSED
test_sample.py:26: test_item_no[11-greater_than_30] FAILED
test_sample.py:26: test_item_no[11-lesser_than_30] PASSED
test_sample.py:26: test_item_no[11-modulo_2] FAILED
test_sample.py:26: test_item_no[33-greater_than_30] PASSED
test_sample.py:26: test_item_no[33-lesser_than_30] FAILED
test_sample.py:26: test_item_no[33-modulo_2] FAILED

====================== 5 failed, 4 passed in 0.03 seconds ======================

See http://asciinema.org/a/6562