CODEWITHSUNDEEP
pythonswigpypipybind11
Alex Lenail
Alex Lenail

Reputation: 14440

How to put a swig/pybind11 C++ project on pypi

I have code wrapped with both pybind11 and swig, but can't seem to find docs about how to properly get that code on pypi so I can pip install my package.

Pybind11 seems to have issues that make it difficult to put code on pypi.

I think what I want is to build binary wheels for Linux & OSX, but I can't find much documentation about this. Some people do this via travis and scripting I guess? Is there a plug and play way to make wheels for all distributions?

Upvotes: 2

Views: 2685

Answers (1)

Roman Miroshnychenko
Roman Miroshnychenko

Reputation: 1564

It is strange that you haven't found any info. distutuis and their bigger brother setuptools provide Extension class that allow to build binary Python-C extension modules before installing them, and any non-Python files can be included in a distribution via MANIFEST.in file. As for Pybind11, it's a header-only library so there should be no difficulties to build a module that depends on it. However, it is better to include Pybind11 headers in your distribution (it's about a dozen not very big .h files), because, as to my knowledge, distutils/setuptools do not support pre-build dependencies for binary modules.

Let's assume that your header files (including Pybind11) are in include dir and your source files are in src dir. Then your setup.py file should look like this:

import os
from setuptools import setup
from setuptools.extension import Extension

this_dir = os.path.dirname(os.path.abspath(__file__))

foo = Extension(
    name='foo',
    include_dirs=[os.path.join(this_dir, 'include')],
    sources=[
        os.path.join(this_dir, 'src', 'foo.cpp'),
        os.path.join(this_dir, 'src', 'bar.cpp')
    ]
)

setup(
    name='foo',
    version='0.0.1',
    author='John Doe',
    description='foo module',
    long_description='blah, blah, blah...',
    url='http://example.com',
    classifiers=[
        # The list of PyPI classifiers
    ],
    ext_modules=[foo],
    zip_safe=False,
    include_package_data=True,
)

You can add other parameters like macro definitions and such. Your MANIFEST.in should look like this:

recursive-include src *.cpp
recursive-include include *.h

Now you can publish your package as described here: https://packaging.python.org I can only note that their recommendation about using twine is valid only for ancient Python distributions with poor HTTPS support.

Now any user can install your package in their environment by typing pip install foo, provided they have installed a C/C++ compiler compatible with their Python version. This is usually no problem on Linux (I have no knowledge of Mac), but may be PITA on Windows. So to simplify installing you can also add pre-compiled wheels to your source distribution as described in the packaging guide above. You can use some CI like Travis or Appveyor (Windows-based CI) to automate compilation of your wheels and publishing them on PyPI. If PyPI has a wheel for the target platform, it is simply unpacked to user's environment. Otherwise, the module is compiled from sources (again, if a compatible compiler is available, otherwise build/install will fail).

I'm not familiar with SWIG but as stated in the documentation, converting SWIG wrappers into C code during compilation is also supported by setuptools natively. SWIG wrapper files are also included via MANIFEST.in.

Things are complicated if your binary Python module depends on some external pre-built libraries, like Boost or OpenSSL. As I said, setuptools does not support pre-build dependencies and on Windows, for example, there is no central repo for binary libraries (although Microsoft is trying to amend the situation by creating vcpkg). In this case your either include everything in your package, or provide statically compiled wheels for as many platforms as possible, or somehow warn potential users that they need to install some prerequisites before pip-installing your binary module.

Upvotes: 13

Related Questions