CODEWITHSUNDEEP
setuptoolspybind11
user3093226
user3093226

Reputation: 43

Distributing pybind11 extension linked to third party libraries

I'm working on a pybind11 extension written in C++ but I'm having a hard time understanding how should it be distributed.

The project links to a number of third party libraries (e.g. libpng, glew etc.).

The project builds fine with CMAKE and it generates a .so file. Now I am not sure what is the right way of installing this extension. The extension seems to work, as if I try copy the file into the python lib directories it is picked up (I can import it, and it works correctly). However, this is clearly not the way to go I think.

I also tried the setuptools route (from https://pybind11.readthedocs.io/en/stable/compiling.html) by creating a setup.py files like this:

import sys

# Available at setup time due to pyproject.toml
from pybind11 import get_cmake_dir
from pybind11.setup_helpers import Pybind11Extension, build_ext
from setuptools import setup
from glob import glob

files = sorted(glob("*.cpp"))

__version__ = "0.0.1"

ext_modules = [
    Pybind11Extension("mylib",
        files,
        # Example: passing in the version to the compiled code
        define_macros = [('VERSION_INFO', __version__)],
        ),
]

setup(
    name="mylib",
    version=__version__,
    author="fab",
    author_email="fab@fab",
    url="https://github.com/pybind/python_example",
    description="mylib",
    long_description="",
    ext_modules=ext_modules,
    extras_require={"test": "pytest"},
    cmdclass={"build_ext": build_ext},
    zip_safe=False,
    python_requires=">=3.7",
)

and now I can build the extension by simply calling

pip3 install

however it looks like all the links are broken because whenever I try importing the extension in Python I get linkage errors, as if setuptools does not link correctly the extension with the 3rd party libs. For instance errors in linking with libpng as in:

>>> import mylib
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /home/fabrizio/.local/lib/python3.8/site-packages/mylib.cpython-38-x86_64-linux-gnu.so: undefined symbol: png_sig_cmp

However I have no clue how to add this link info to setuptools, and don't even know if that's possible (it should be the setuptools equivalent of CMAKE's target_link_libraries). I am really at a loss after weeks of reading documentation, forum threads and failed attempts. If anyone is able to point me in the right way or to clear some of the fog it would be really appreciated!

Thanks! Fab

Upvotes: 0

Views: 1111

Answers (1)

Milan Š.
Milan Š.

Reputation: 1628

/home/fabrizio/.local/lib/python3.8/site-packages/mylib.cpython-38-x86_64-linux-gnu.so: undefined symbol: png_sig_cmp

This line pretty much says it clearly. Your local shared object file .so can't find the libpng.so against which it is linked.

You can confirm this by running:

ldd /home/fabrizio/.local/lib/python3.8/site-packages/mylib.cpython-38-x86_64-linux-gnu.so

There is no equivalent of target_link_libraries() in setuptools. Because that wouldn't make any sense. The library is already built and you've already linked it. This is your system more or less telling you that it can't find the libraries it needs. And those most likely need to be installed.

This is also one of the reasons why Linux distributions provide their own package managers and why you should use the developer packages provided by said distributions.

So how do you fix this? Well your .so file needs to find the other .so files against which you linked to understand how this works I will refer you to this link.

My main guess is based on the fact that when you manually copy the files it works - That during the build process you probably specify the rpath to a local directory. Hence what you most likely need to do is specify to your setuptools that it needs to copy those files when installing.

Upvotes: 1

Related Questions