CODEWITHSUNDEEP
pythonpytest
Alex Okrushko
Alex Okrushko

Reputation: 7372

Py.test loop through the tests repeatedly

I'm trying to repeat the tests N number of times (same tests that are collected).

Why: By doing do I want to see if the speed of the tests decreases or I can collect the "average time" of one parameter then pass the other parameter and get the "average time" again.

My understanding is to use def pytest_runtestloop() hook, however I have troubles with it.

Here is my code for the hook:

def pytest_runtestloop(session):
    repeat = int(session.config.option.repeat)
    assert isinstance(repeat, int), "Repeat must be an integer"
    for i in range(repeat): #@UnusedVariable                      
        session.config.pluginmanager.getplugin("main").pytest_runtestloop(session)

    return True

The problem is that it the "setups" are run only the first time: For example:

class TestSomething(object):

    @classmethod
    @pytest.fixture(scope = "class", autouse = True)
    def setup(self):
        //setup function

    def test_something(self):
        //test function

Here setup will be called during the first cycle only, and test_something would be called both times if I set session.config.option.repeat to 2

What am I doing wrong? Is there better approach?

Upvotes: 3

Views: 3149

Answers (1)

hpk42
hpk42

Reputation: 23561

It seems pytest-2.3.4 is keeping some state internally that keeps fixtures from running again. pytest_runtest_loop was not written with your use case in mind. You may file a "bug" issue about it and we can see what we can do to fix it. (I quickly looked but couldn't immediately see what was going wrong, requires more exploring).

Upvotes: 2

Related Questions