CODEWITHSUNDEEP
pytest
Gérard Depardieu
Gérard Depardieu

Reputation: 28

How do I call a test with fixtures in a different process

I'm writing tests for a library built on pytorch, and many of the tests need to exercise torch.distributed by running the same function body on different processes and communicating with each other (e.g. using torch.distributed.all_reduce).

I currently have a decorator which wraps a test like so:

import inspect

@wrap()
def foo():
    assert true == false

def wrap():
    def fixer(function)
        def replacement():
            torch.distributed.mp.spawn(function, nprocs=2)

        setattr(inspect.getmodule(function), "test_" + function.__name__, replacement)
    return fixer

But this doesn't work with pytest fixtures, and I would like to be able to use fixtures. I tried declaring my own pytest_runtest_call/pytest_runtest_protocol but the test item isn't pickle-able so I'm back to square one. Is there some easy way to do this that minimizes the boilerplate for each test?

Currently tests that need fixtures call mp.spawn on another function which contains the actual test body.

Upvotes: 1

Views: 295

Answers (1)

nutic
nutic

Reputation: 457

I guess the problem is not in pytorch or different process but simply in losing the function signature and parameter passing in wrap

def wrap():
    def fixer(function):
        paramertes = inspect.signature(function).parameters
        @wraps(function)  # this will make pytest pass fixtures to the wrapped function  
        def replacement(*args, **kwargs):
            assert args == tuple()
            args = tuple(kwargs[p] for p in paramertes)  # converting named parameters to positional parameters to pass to `spawn`
            torch.distributed.mp.spawn(function, args=args, nprocs=2)

        setattr(inspect.getmodule(function), "test_" + function.__name__, replacement)
    return fixer


@fixture()
def d():
    return 1


@wrap()
def foo(d):
    assert d == 1

Upvotes: 1

Related Questions