py.test to test Cython C API modules

Question

I'm trying to set up unit tests for a Cython module to test some functions that do not have python interface. The first idea was to check if .pyx files could directly be used by py.test's test runner, but apparently it only scans for .py files.

Second idea was to write the test_* methods in one Cython module, which could then be imported into a plain .py file. Let say we have a foo.pyx module with the contents we want to test:

cdef int is_true():
    return False

then a test_foo.pyx module that uses the C API to test the foo module:

cimport foo

def test_foo():
    assert foo.is_true()

and then import these in a plain cython_test.py module that would just contain this line:

from foo_test import *

The py.test test runner does find test_foo this way, but then reports:

/usr/lib/python2.7/inspect.py:752: in getargs
    raise TypeError('{!r} is not a code object'.format(co))
E   TypeError: <built-in function test_foo> is not a code object

Is there any better way to test Cython C-API code using py.test?

Answer 1

So, in the end, I managed to get py.test to run tests directly from a Cython-compiled .pyx files. However, the approach is a terrible hack devised to make use of py.test Python test runner as much as possible. It might stop working with any py.test version different from the one I prepared the hack to work with (which is 2.7.2).

First thing was to defeat py.test's focus on .py files. Initially, py.test refused to import anything that didn't have a file with .py extension. Additional problem was that py.test verifies whether the module's __FILE__ matches the location of the .py file. Cython's __FILE__ generally does not contain the name of the source file though. I had to override this check. I don't know if this override breaks anything—all I can say is that the tests seem to run well, but if you're worried, please consult your local py.test developer. This part was implemented as a local conftest.py file.

import _pytest
import importlib


class Module(_pytest.python.Module):
    # Source: http://stackoverflow.com/questions/32250450/
    def _importtestmodule(self):
        # Copy-paste from py.test, edited to avoid throwing ImportMismatchError.
        # Defensive programming in py.test tries to ensure the module's __file__
        # matches the location of the source code. Cython's __file__ is
        # different.
        # https://github.com/pytest-dev/pytest/blob/2.7.2/_pytest/python.py#L485
        path = self.fspath
        pypkgpath = path.pypkgpath()
        modname = '.'.join(
            [pypkgpath.basename] +
            path.new(ext='').relto(pypkgpath).split(path.sep))
        mod = importlib.import_module(modname)
        self.config.pluginmanager.consider_module(mod)
        return mod

    def collect(self):
        # Defeat defensive programming.
        # https://github.com/pytest-dev/pytest/blob/2.7.2/_pytest/python.py#L286
        assert self.name.endswith('.pyx')
        self.name = self.name[:-1]
        return super(Module, self).collect()


def pytest_collect_file(parent, path):
    # py.test by default limits all test discovery to .py files.
    # I should probably have introduced a new setting for .pyx paths to match,
    # for simplicity I am hard-coding a single path.
    if path.fnmatch('*_test.pyx'):
        return Module(path, parent)

Second major problem is that py.test uses Python's inspect module to check names of function arguments of unit tests. Remember that py.test does that to inject fixtures, which is a pretty nifty feature, worth preserving. inspect does not work with Cython, and in general there seems to be no easy way to make original inspect to work with Cython. Nor there is any other good way to inspect Cython function's list of arguments. For now I decided to make a small workaround where I'm wrapping all test functions in a pure Python function with desired signature.

In addition to that, it seems that Cython automatically puts a __test__ attribute to each .pyx module. The way Cython does that interferes with py.test, and needed to be fixed. As far as I know, __test__ is an internal detail of Cython not exposed anywhere, so it should not matter that we're overwriting it. In my case, I put the following function into a .pxi file for inclusion in any *_test.pyx file:

from functools import wraps


# For https://github.com/pytest-dev/pytest/blob/2.7.2/_pytest/python.py#L340
# Apparently Cython injects its own __test__ attribute that's {} by default.
# bool({}) == False, and py.test thinks the developer doesn't want to run
# tests from this module.
__test__ = True


def cython_test(signature=""):
    ''' Wrap a Cython test function in a pure Python call, so that py.test
    can inspect its argument list and run the test properly.

    Source: http://stackoverflow.com/questions/32250450/'''
    if isinstance(signature, basestring):
        code = "lambda {signature}: func({signature})".format(
            signature=signature)

        def decorator(func):
            return wraps(func)(eval(code, {'func': func}, {}))

        return decorator

    # case when cython_test was used as a decorator directly, getting
    # a function passed as `signature`
    return cython_test()(signature)

After that, I could implement tests like:

include "cython_test_helpers.pxi"
from pytest import fixture

cdef returns_true():
    return False

@cython_test
def test_returns_true():
    assert returns_true() == True

@fixture
def fixture_of_true():
    return True

@cython_test('fixture_of_true')
def test_fixture(fixture_of_true):
    return fixture_of_true == True

If you decide to use the hack described above, please remember to leave yourself a comment with a link to this answer—I'll try to keep it updated in case better solutions are available.