How to use Cython with pytest?

Question

The goal is to use the pytest unit test framework for a Python3 project that uses Cython. This is not a plug-and-play thing, because pytest by default is not able to import the Cython modules.

One unsuccessful solution would be to use the pytest-cython plugin, but it simply does not work for me:

> py.test --doctest-cython
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: --doctest-cython
  inifile: None
  rootdir: /censored/path/to/my/project/dir

To verify that I have the package installed:

> pip freeze | grep pytest-cython
pytest-cython==0.1.0

UPDATE: I'm using PyCharm and it seems that it is not using my pip-installed packages but rather uses a custom(?) pycharm repository for packages used by my project. Once I added pytest-cython to that repository, the command runs but strange enough it doesn't recognize the Cython module anyway, although the package/add-on is specifically designed for that purpose:

> pytest --doctest-cython
Traceback:
tests/test_prism.py:2: in <module>
    from cpc_naive.prism import readSequence, processInput
cpc_naive/prism.py:5: in <module>
    from calculateScore import calculateScore, filterSortAlphas, 
calculateAlphaMatrix_c#, incrementOverlapRanges  # cython code
E   ImportError: No module named 'calculateScore'

Another unsuccessful solution I got here is to use pytest-runner, but this yields:

> python3 setup.py pytest
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'pytest'

UPDATE:

I first had forgotten to add setup_requires=['pytest-runner', ...] and tests_require=['pytest', ...] to the setup script. Once i did that, I got another error:

> python3 setup.py pytest
Traceback (most recent call last):
  File "setup.py", line 42, in <module>
    tests_require=['pytest']
(...)
AttributeError: type object 'test' has no attribute 'install_dists'

UPDATE 2 (setup.py):

from distutils.core import setup
from distutils.extension import Extension

from setuptools import find_packages
from Cython.Build import cythonize
import numpy

try:  # try to build the .c file
    from Cython.Distutils import build_ext
except ImportError:  # if the end-user doesn't have Cython that's OK; you should have shipped the .c files anyway.
    use_cython = False
else:
    use_cython = True

cmdclass = {}
ext_modules = []

if use_cython:
    ext_modules += [
        Extension("cpc_naive.calculateScore", ["cpc_naive/calculateScore.pyx"],
                  extra_compile_args=['-g'],   # -g for debugging
                  define_macros=[('CYTHON_TRACE', '1')]),
    ]
    cmdclass.update({'build_ext': build_ext})
else:
    ext_modules += [
        Extension("cpc_naive.calculateScore", ["cpc_naive/calculateScore.c"],
                  define_macros=[('CYTHON_TRACE', '1')]),  # compiled C files are stored in /home/pdiracdelta/.pyxbld/
    ]

setup(
    name='cpc_naive',
    author=censored,
    author_email=censored,
    license=censored,
    packages=find_packages(),
    cmdclass=cmdclass,
    ext_modules=ext_modules,
    install_requires=['Cython', 'numpy'],
    include_dirs=[numpy.get_include()],
    setup_requires=['pytest-runner'],
    tests_require=['pytest']
)

UPDATE 3 (partial fix): As suggested by @hoefling I downgraded pytest-runner to a version <4 (in fact 3.0.1) and this resolves the error in update 1, but now I get the same Exception as with the pytest-cython solution:

E   ImportError: No module named 'calculateScore'

It just doesn't seem to recognize the module. Perhaps this is due to some absolute/relative import mojo I don't understand.

How can I use pytest with Cython? How can I discover why these methods aren't working and then fix it?

FINAL UPDATE: After taking both the original problem and the question Updates into consideration (thanks @hoefling for solving these issues!), this question is now reduced to the question of:

why can pytest no import the Cython module calculateScore, even though running the code just with python (no pytest) works just fine?

Answer 1

As @hoefling suggested, one should use pytest-runner version <0.4 to avoid the

AttributeError: type object 'test' has no attribute 'install_dists'

To then answer the actual and final question (in addition to partial, off-topic, user-specific fixes added to the question post itself) of why pytest cannot import the Cython module calculateScore, even though running the code just with python (no pytest) works just fine: that remaining issue is solved here.