CODEWITHSUNDEEP
pythonpackagesetuptoolsdistutils
Kit
Kit

Reputation: 31533

Distribute a Python package with a compiled dynamic shared library

How do I package a Python module together with a precompiled .so library? Specifically, how do I write setup.py so that when I do this in Python

>>> import top_secret_wrapper

It can easily find top_secret.so without having to set LD_LIBRARY_PATH?

In my module development environment, I have the following file structure:

.
├── top_secret_wrapper
│   ├── top_secret.so
│   └── __init__.py
└── setup.py

Inside __init__.py, I have something like:

import top_secret

Here's my setup.py

from setuptools import setup, Extension

setup(
    name = 'top_secret_wrapper',
    version = '0.1',
    description = 'A Python wrapper for a top secret algorithm',
    url = None,
    author = 'James Bond',
    author_email = '[email protected]',
    license = 'Spy Game License',
    zip_safe = True,
)

I'm sure my setup.py is lacking a setting where I specify the location of top_secret.so, though I'm not sure how to do that.

Upvotes: 42

Views: 19707

Answers (4)

Yan QiDong
Yan QiDong

Reputation: 4441

As is mentioned in setupscript.html#installing-package-data:

setup(
    ...
    package_data={'top_secret_wrapper': ['top_secret.so']},
)

Upvotes: 6

souch
souch

Reputation: 372

I managed to bundle a .so (that has other .so dependancies) in its python package directory like this:

  • build the mypackage_bindings.cpython-310-x86_64-linux-gnu.so containing Python bindings of C++ using pybind11 and cmake.
  • using this minimal python package dir infrastructure:
setup.cfg
setup.py
README.md
mypackage/__init__.py
mypackage/mypackage_bindings.cpython-310-x86_64-linux-gnu.so
mypackage/some_deps.so
  • set rpath of mypackage_bindings.so and its .so dependancies to $ORIGIN using these commands on linux (so that the linker will search the deps in the same .so dir):
patchelf --set-rpath '$ORIGIN' mypackage_bindings.cpython-310-x86_64-linux-gnu.so
patchelf --set-rpath '$ORIGIN' some_deps.so
  • put in mypackage/__init__.py:
import os
import sys

cur_file_dir = os.path.dirname(os.path.realpath(__file__))

# add current file directory so that mypackage_bindings.so is found by python
sys.path.append(cur_file_dir)

# set current file directory as working dir so that mypackage_bindings.so dependancies
# will be found by the linker (mypackage_bindings.so and its deps RPATH are set to $ORIGIN)
os.chdir(cur_file_dir)

# load every symbols of mypackage_bindings into upper mypackage module
from mypackage_bindings import *
  • put in setup.py:
from setuptools import setup

setup(
    name='mypackage',
    packages=['mypackage'],
    package_dir={'mypackage': 'mypackage'},
    package_data={'mypackage': ['*.so', 'lib*']},
    description='Provides mypackage to python users',
    version='0.1',
    url='https://yo.com',
    author='truc muche',
    author_email='[email protected]',
    keywords=['pip', 'mypackage']
    )
  • from the minimal python package dir, launch this command to create the python package:

python3 setup.py sdist

That way, there is no need to set LD_LIBRARY_PATH variable, the .so are installed in the pythonX.X/site-packages/mypackage/ directory.

Upvotes: 2

Ibolit
Ibolit

Reputation: 9730

What I ended up doing is: 

setup(
    name='py_my_lib',
    version=version,  # specified elsewhere
    packages=[''],
    package_dir={'': '.'},
    package_data={'': ['py_my_lib.so']},
)

This way I get to import the lib by its name, and don't have another level of nestedness:

import py_my_lib

and not

from py_my_lib_wrapper import py_my_lib

Upvotes: 17

renefritze
renefritze

Reputation: 2243

If that library should also be compiled during install you can describe this as an extension module. If you just want to ship it add it as package_data

Upvotes: 3

Related Questions