Prepare package with Cython extension to publish on PyPI

Question

I'm going to create C library and I would like to create Python wrapper using Cython.

Right now I have mylib.a file compiled and bundled (C files) and I want to wrap methods from my library in Cython.

I successfully created .pyx and .pxd files and I can build an extension using python setup.py build_ext command. My problem appears when I try to publish it on PyPI. If I run my setup.py, create a wheel and publish it I can download it from PyPI - but I can't run import mylib.

I read a lot online tutorials for this. Few of them shown that theirs C code compiled on user side. My codebase will be constructed from many files and I would prefer to supply already build C code as .a file.

My file structure (only important files):

/ lib
    - mylib.a
    - *.h files (for mylib.a)
- setup.py
/ mylib
    - mylib.pyx
    - cmylib.pxd
    - __init__.py

My __init__.py file (after importing package downloaded from PyPI it throws errors here):

from . import mylib

My setup.py (only important parameters - in my opinion):

setup(name='mylib'
      packages=['mylib'],
      ext_modules = [Extension(
          name="mylib", 
          sources=["mylib/mylib.bycython.c"], 
          include_dirs = [numpy.get_include(), "lib/"],
          extra_objects=["lib/mylib.a"])],
        "build_ext": build_ext
      }
)

(I build mylib.bycython.c from mylib.pyx using cython command before python setup.py build_ext. According to this article it will make package installation faster and will not require user to have the same Cython version.)

It maybe worth mentioning - after building my package I get .so file for mylib. If I copy it to mylib/ directory then from parent directory I can import mylib and access my methods using mylib.mylib.say_hello(). However it doesn't work on package installed from PyPI (on other PC) neither I don't want to use mylib.mylib.

If I should provide more info - let me know.

Edit:

My real project on GitHub: https://github.com/franiis/statr-python.

I want to successfully run say_hello() method from statr.pyx (other methods probably will not work).

I know code has some problems, but I want to first have a working core to fix and update everything. To build the project use build_script.sh. upload_script.sh creates a wheel and publishes it.

Answer 1

If the only thing to install in your project is a single extension module, there's no need to define any packages. You could simply install your extension as a top-level module:

from setuptools import setup, Extension
import numpy


setup(
    name = 'mylib',
    ...
    ext_modules = [
        Extension(
            name = 'mylib',
            sources = ['mylib/mylib.bycython.c'],
            include_dirs = [numpy.get_include(), 'lib/'],
            # cross-platform library names (without the "lib" prefix on Unix-likes):
            libraries = ['lib/mylib']
        )
    ],
    # legacy option for setup.py-style projects to install an unpacked distribution
    zip_safe = False,
    ...
)

and use it directly with import mylib.

Also note that in most cases there's no need to use Cython package in your setup.py. setuptools does this for you and accepts the .pyx files as exten­sion sources.

Answer 2

There are several issues in the setup.py.

• The linked to library is not being compiled by setup.py. This means it must be compiled manually.
• The linked to library is a pre-compiled statically linked archive. This is not portable and cannot be distributed to anything other than the exact versions of GCC/glibc used to build it.
• The cython extension code is not being built by setup.py. Changes to .pyx/.pxd will not be reflected in the package.

Try something like:

from setuptools import setup, find_packages, Extension
from Cython.Build import cythonize, build_ext

extensions = [Extension(
          name="statr._ext", 
          sources=["statr/_ext.pyx", "lib/mylib.c"],
          depends="lib/mylib.h",
          include_dirs=[numpy.get_include(), "lib/"]
]

setup(name='statr'
      packages=find_packages(),
      ext_modules=cythonize(extensions),
      build_ext: build_ext
      }
)

With the above, the name of the extension will be statr._ext. The name of the pyx should be _ext.pyx.

mylib.c will be built and linked to the extension by setup.py. Any additional compiler directives required by mylib.c should be added to the extension.

The package name will be statr. You should import anything from _ext that you want to be available from the top level statrd module in its __init__.py, for example

from ._ext import my_cython_function