CODEWITHSUNDEEP
pythonpython-3.xpipsetup.pypython-wheel
Sc4v
Sc4v

Reputation: 358

python: create wheel from existing c extension

Our build pipeline spits out an *.so / *.pyb python extension through pybind11. As a step in the pipeline I need to package the extension as a wheel for easy distribution through pip. I am trying to come up with a setup.py that takes the existing library and does not rely on re-compiling the binaries through the setup.py (so I really don't want this). It would require a major rewrite of the devops scripts.

When having a folder structure such as:

setup.py
my_module.cpython-39-darwin.so

A very basic setup.py can create a functioning wheel (python setup.py bdist_wheel):

setup(
    name = 'my_module', 
    version='0.9.102', 
    packages=["."],
    package_data={'': ['*.so', '*.pyi']},
)

Unfortunately, the wheel is missing the important python tag and platform name, etc: my_module-0.9.102-py3-none-any.whl vs. my_module-0.9.102-cp39-cp39-macosx_10_13_x86_64.whl

Setting --python-tag and --plat-name works, setting --py-limited-api does not, however.

Through research I found that overwriting the distclass adds the correct tag again, but the Root-Is-Purelib is set back to false. This, unfortunately, creates a broken wheel when installing through pip as it puts the binary in a my_module-0.9.102.data/purelib folder...

Overwriting the is_pure seems to be ignored also:

from setuptools import setup, find_packages, Distribution

class BinaryDistribution(Distribution):
    def is_pure(self):
        return True

    def has_ext_modules(foo):
        return True

setup(
    name = 'my_module', 
    version='0.9.102', 
    packages=["."],
    package_data={'': ['*.so', '*.pyi']},
    distclass=BinaryDistribution
)

What else can I do to wrap my pre-compiled python libraries to wheels for distribution without rewriting lots of the build pipeline?

Upvotes: 3

Views: 1440

Answers (1)

Sc4v
Sc4v

Reputation: 358

Not a perfect solution but a functional wheel is created when using the wheel module instead:

from setuptools import setup

setup(name='my_module',
      packages=['my_module'],
      package_data={'': ['*.so', '*.pyi']},
      )

create wheel with:

pip install wheel
pip wheel .

Upvotes: 1

Related Questions