CODEWITHSUNDEEP
pythonpytest
Jonathan
Jonathan

Reputation: 7551

How to write pytest tests that run in 2 stages, each stage sequentially?

I'm writing a bunch of tests in pytest, many of which take the form (pseudo-code):

def test1(self, ...):
    expected_events = do_thing1_that_should_generate_events(...)
    assert wait_for_events_to_appear_in_ui(expected_events)

The events take a few minutes to appear in the UI, so wait_for_events_to_appear_in_ui polls the UI for them, with a timeout. This works, but every such test runs for a few minutes, so total run time would be long.

So, I want to run all the do_thingX actions first, and then run all the validations. I thought about adding the expected events to some list, and having a final test that verifies all of them - but that means that if do_thing1, pytest would not report test1 as failing, but the final test - which is sub-optimal.

Ideally, the tests would look something like:

def test1(self, ...):
    expected_events = do_thing1_that_should_generate_events(...)
    # some kind of magic that works like yield?
    assert wait_for_events_to_appear_in_ui(expected_events)

How can I achieve that?

Upvotes: 1

Views: 1614

Answers (1)

Cyphase
Cyphase

Reputation: 12002

First take a look at pytest-xdist, which is for parallelizing tests among multiple processes, among other things. I don't know how well that'll work for your case though. Read on for a more specific solution.

You need to manage the asynchronous running of these tests inside a single test function, which you hinted at in your question. Maybe you do it with sync code, maybe you use an async framework; that's orthogonal to this. I'm going to assume you can manage this on your own, and in any event it's not a pytest-specific thing.

To get around the problem of a blanket pass/fail inside the one pytest test function, you need subtests. pytest-subtests allows you to write subtests inside a single test function by providing a subtests fixture (it also supports subTest() from unittest).

Here's the provided example:

def test(subtests):
    for i in range(5):
        with subtests.test(msg="custom message", i=i):
            assert i % 2 == 0

Output of pytest:

λ pytest .tmp\test-subtest.py
======================== test session starts ========================
...
collected 1 item

.tmp\test-subtest.py .F.F..                                    [100%]

============================= FAILURES ==============================
____________________ test [custom message] (i=1) ____________________

    def test(subtests):
        for i in range(5):
            with subtests.test(msg='custom message', i=i):
>               assert i % 2 == 0
E               assert (1 % 2) == 0

.tmp\test-subtest.py:4: AssertionError
____________________ test [custom message] (i=3) ____________________

    def test(subtests):
        for i in range(5):
            with subtests.test(msg='custom message', i=i):
>               assert i % 2 == 0
E               assert (3 % 2) == 0

.tmp\test-subtest.py:4: AssertionError
================ 2 failed, 1 passed in 0.07 seconds =================

The failed/passed numbers are a bit counter-intuitive at first glance. It looks like we have five subtests, of which two should fail; and indeed we have two reported failures. But we only have one passing test, not three. That one pass is test_func() itself; since the with block is catching the asserts inside, test_func() isn't failing. The other three subtests didn't fail, and therefore were not reported at all – although we do see them represented in the progress dots.

In summary: manage the tests yourself, and use pytest-subtests to get individual reporting of the failures.

Upvotes: 2

Related Questions