CODEWITHSUNDEEP
pythonrustpytestpyo3
Anton Shpigunov
Anton Shpigunov

Reputation: 182

Rust libraries made with PyO3 and rust-cpython won't import when running pytest

While following various tutorials on compiling Rust libraries for further import from Python (I've tried both PyO3 and rust-cpython), I've been able to build a simple libary and successfully import it from my main.py and interactive Python shells.

However, when trying to test and benchmark pure Python vs. my Rust library, I consistently get import errors:

_____________________________________________ ERROR collecting main.py _____________________________________________
ImportError while importing test module '/Users/xxx/pyo3/sumlib/main.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/xxx/.pyenv/versions/3.8.2/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
main.py:1: in <module>
    import sumlib
E   ModuleNotFoundError: No module named 'sumlib'

Again, the library imports just fine from the main python file and the interactive shells like IPython.

I'm on macOS Big Sur, Python 3.8 installed using pyenv. Here is my Cargo.toml:

[package]
name = "sumlib"
version = "0.1.0"
edition = "2018"

[lib]
name = "sumlib"
# "cdylib" is necessary to produce a shared library for Python to import from.
#
# Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
# to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
# crate-type = ["cdylib", "rlib"]
crate-type = ["cdylib", "rlib"]

[dependencies.pyo3]
version = "0.14.2"
features = ["extension-module"]

Everything else is very trivial, from the PyO3 and rust-cpython, to the same result.

Upvotes: 2

Views: 1985

Answers (1)

Emerson Harkin
Emerson Harkin

Reputation: 917

tl/dr: Invoke your tests using python -m pytest instead of pytest.

It sounds like you haven't installed your Python extension module (eg, using pip install) and are instead relying on it being inside your working directory. This would allow you to import it using an interpreter or main.py launched via python -m,

$ tree .
.
├── bin
│   ├── __init__.py
│   └── sumlib.py
└── examples
    └── main.py

2 directories, 3 files
$ ipython --no-banner

In [1]: from bin import sumlib

In [2]: exit()

$ python -m examples.main
Successfully imported sumlib!

but not using pytest examples/main.py or python examples/main.py.

$ pytest examples/main.py
======================================================= test session starts =======================================================
platform darwin -- Python 3.8.3, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: ...
plugins: typeguard-2.10.0
collected 0 items / 1 error

============================================================= ERRORS ==============================================================
________________________________________________ ERROR collecting examples/main.py ________________________________________________
ImportError while importing test module '.../examples/main.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.../python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
examples/main.py:5: in <module>
    from bin import sumlib
E   ModuleNotFoundError: No module named 'bin'
...
$ python examples/main.py
Traceback (most recent call last):
  File "examples/main.py", line 5, in <module>
    from bin import sumlib
ModuleNotFoundError: No module named 'bin'

This is because ipython and python -m both add the current directory, which contains your extension module, to Python's sys.path, but pytest examples/main.py and python examples/main.py don't. (See the pytest docs.) You can solve this by either installing your extension module or running your tests using python -m pytest instead of pytest.

Upvotes: 1

Related Questions