CODEWITHSUNDEEP
pythonunit-testingrustmaturin
davidA
davidA

Reputation: 13664

How to run Rust library unit tests with Maturin?

I'm building a custom Python module in Rust, with maturin.

I am using PyEnv, with Python 3.12.2, installed with env PYTHON_CONFIGURE_OPTS="--enable-shared" and --keep so the Python sources are available. I have a venv and I'm using maturin develop to build my library and update the venv as I need to.

I have some unit tests in my lib.rs file, and usually I'd just run cargo test to run them.

But within the Maturin-managed environment, I get linker errors:

error: linking with `cc` failed: exit status: 1
...
  = note: /usr/bin/ld: .../target/debug/deps/libpyo3-c286c2d4bbe22ea2.rlib(pyo3-c286c2d4bbe22ea2.pyo3.7555584b645de9e8-cgu.01.rcgu.o): in function `pyo3_ffi::object::Py_DECREF':
          $HOME/.cargo/git/checkouts/pyo3-a22e69bc62b9f0fd/da24f0c/pyo3-ffi/src/object.rs:597: undefined reference to `_Py_Dealloc'

I was able to work around this with the following two methods:

  1. Using RUSTFLAGS and LD_LIBRARY_PATH / rpath:
$ export RUSTFLAGS="-C link-arg=-Wl,-rpath,.../pyenv.git/versions/3.12.2/lib -C link-arg=-L.../pyenv.git/versions/3.12.2/lib -C link-arg=-lpython3.12"
  1. Or creating .cargo/config.toml:
[target.'cfg(all())']
rustflags = [
    "-C", "link-arg=-Wl,-rpath,.../pyenv.git/versions/3.12.2/lib",
    "-C", "link-arg=-L.../pyenv.git/versions/3.12.2/lib",
    "-C", "link-arg=-lpython3.12",
]

I've snipped the paths for brevity/privacy.

Both of these methods are doing the same thing - providing an explicit linker path, library to link against, and run-time library path. This works, but it feels wrong.

I wasn't able to find a maturin test or equivalent, and it doesn't seem right that I have to manually specify the linker arguments to cargo / rustc when I have maturin right there to do that for me.

Is there a good way to do this with maturin?

EDIT: In the PyO3 docs I found the Testing section, which recommends converting this part of Cargo.toml:

[dependencies]
pyo3 = { git = "https://github.com/pyo3/pyo3", features = ["extension-module"] }

Into this:

[dependencies.pyo3]
git = "https://github.com/pyo3/pyo3"

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

And then running cargo test --no-default-features. This does in fact resolve the linker errors, but it still has trouble locating the library at runtime:

     Running unittests src/lib.rs (target/debug/deps/pyo3_example-cc3941bd091dc64e)
.../target/debug/deps/pyo3_example-cc3941bd091dc64e: error while loading shared libraries: libpython3.12.so.1.0: cannot open shared object file: No such file or directory

This is resolved with setting LD_LIBRARY_PATH to .../pyenv.git/versions/3.12.2/lib, so it goes some way to helping, but it's not quite sufficient.

Upvotes: 7

Views: 741

Answers (1)

cadolphs
cadolphs

Reputation: 9617

I have pretty much the same setup: Python module with PyO3 and maturin. I run cargo test no problem. Maybe it's because in my cargo.toml I have this section:

[lib]
name = "module_name_whatever"
crate-type = ["cdylib", "lib"]

EDIT: To help OP troubleshoot, here's my full Cargo.toml

[package]
name = "package_name"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[lib]
name = "package_name"
crate-type = ["cdylib", "lib"]

[dependencies.pyo3]
version = "0.20.0"
# "abi3-py38" tells pyo3 (and maturin) to build using the stable ABI with minimum Python version 3.8
features = ["abi3-py38"]

Upvotes: 0

Related Questions