Building a ctypes-"based" C library with distutils

Question

Following this recommendation, I have written a native C extension library to optimise part of a Python module via ctypes. I chose ctypes over writing a CPython-native library because it was quicker and easier (just a few functions with all tight loops inside).

I've now hit a snag. If I want my work to be easily installable using distutils using python setup.py install, then distutils needs to be able to build my shared library and install it (presumably into /usr/lib/myproject). However, this not a Python extension module, and so as far as I can tell, distutils cannot do this.

I've found a few references to people other people with this problem:

• Someone on numpy-discussion with a hack back in 2006.
• Somebody asking on distutils-sig and not getting an answer.
• Somebody asking on the main python list and being pointed to the innards of an existing project.

I am aware that I can do something native and not use distutils for the shared library, or indeed use my distribution's packaging system. My concern is that this will limit usability as not everyone will be able to install it easily.

So my question is: what is the current best way of distributing a shared library with distutils that will be used by ctypes but otherwise is OS-native and not a Python extension module?

Feel free to answer with one of the hacks linked to above if you can expand on it and justify why that is the best way. If there is nothing better, at least all the information will be in one place.

Answer 1

Update: This answer basically works but has deployment issues (missing ctypes extensions when running setuptools install commands) and is quite complicated. See my other answer instead, which ports memeplex's answer from distutils to setuptools.

---

In 2024, distutils is deprecated, but the question is still relevant for its direct successor setuptools. The setup command in setuptools has an option libraries to build libraries which are not extension modules. This option is designed to only build libraries which are needed as a dependency for extension modules (statically linked and not included as library file in binary distributions). However, the corresponding setuptools command class can be changed to produce shared libraries (which are loadable by ctypes) and include them in binary distributions. This is the relevant code in setup.py:

import os

from setuptools import setup
from setuptools.command.build_clib import build_clib


class BuildSharedLibraryCommand(build_clib):
    """
    Build a shared library.

    This is a modified version of the [build_clib class in
    setuptools](https://github.com/pypa/setuptools/blob/v75.1.0/setuptools/command/build_clib.py#L16-L110).
    """

    def initialize_options(self):
        super().initialize_options()
        # Defining `self.inplace` is important, otherwise the `editable_wheel` command invoked by `pip install -e` will
        # not set `self.inplace` to `True`.
        self.inplace = False

    def finalize_options(self):
        super().finalize_options()
        # `self.build_clib` is set to a temporary directory in the base class. However, we want to build the shared
        # library in the actual build directory, so reset it here.
        self.build_clib = None
        self.set_undefined_options(
            "build",
            ("build_lib", "build_clib"),
        )
        # self.inplace is set by the `editable_wheel` command invoked by `pip install -e`. However, double check and
        # get the setting from the `build_ext` if needed.
        self.set_undefined_options(
            "build_ext",
            ("inplace", "inplace"),
        )

    def build_libraries(self, libraries):
        for lib_name, build_info in libraries:
            sources = sorted(list(build_info.get("sources")))

            expected_objects = self.compiler.object_filenames(
                sources,
                output_dir=self.build_temp,
            )

            macros = build_info.get("macros")
            include_dirs = build_info.get("include_dirs")
            cflags = build_info.get("cflags")
            self.compiler.compile(
                sources,
                output_dir=self.build_temp,
                macros=macros,
                include_dirs=include_dirs,
                extra_postargs=cflags,
                debug=self.debug,
            )

            if self.inplace:
                output_dir = os.path.dirname(sources[0])
            else:
                output_dir = os.path.join(self.build_clib, os.path.dirname(sources[0]))
            link_libraries = build_info.get("link_libraries")
            library_dirs = build_info.get("library_dirs")
            lflags = build_info.get("lflags")
            if platform.system() == "Windows":
                if lflags is None:
                    lflags = []
                # The `link_shared_lib` command below does not include the `/DLL` flag, so add it here.
                if "/DLL" not in lflags:
                    lflags.append("/DLL")
            # Link the object files together into a shared library and store it in the package where the first source
            # file is located.
            self.compiler.link_shared_lib(
                expected_objects,
                lib_name,
                output_dir=output_dir,
                libraries=link_libraries,
                library_dirs=library_dirs,
                extra_postargs=lflags,
                debug=self.debug,
            )

    def get_library_names(self):
        # `build_clib` was actually designed to build a static lib which is linked to all extension modules. However,
        # this class modifies that behavior to build a shared library which shall NOT be linked to the extension
        # modules. The extension module link code queries `build_clib` libraries by calling `get_library_names()`.
        # Therefore, clear the library list, so no unwanted libraries will be linked.
        return []


setup(
    cmdclass={"build_clib": BuildSharedLibraryCommand},
    libraries=[
        (
            "algorithm",
            {
                "sources": ["myproject/calculation/algorithm.c"],
                "include_dirs": [],
                "library_dirs": [],
                "link_libraries": [],
            },
        )
    ],
)

In the setup call the cmdclass parameter is used to override the default build_clib command. The libraries parameter declares all libraries that need to be built. On Linux, this would produce a libalgorithm.so which is located in the the same directory as the first source file.

Answer 2

This is a modernized version of @memeplex's answer for setuptools since the direct usage of distutils is deprecated:

import os
import platform

from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext as build_ext_orig


class build_ext(build_ext_orig):
    def get_export_symbols(self, ext):
        # For CTypes extensions, do not require to export a `PyInit_` function
        if isinstance(ext, CTypes):
            return ext.export_symbols
        return super().get_export_symbols(ext)

    def get_ext_filename(self, fullname):
        # For CTypes extensions, force to use the default system prefix and extension for shared libraries.
        # This avoids file extensions like `.cpython-312-x86_64-linux-gnu.so`.
        if isinstance(self.ext_map[fullname], CTypes):
            shared_lib_prefix = {
                "Windows": "",
            }.get(platform.system(), "lib")
            shared_lib_ext = {
                "Darwin": ".dylib",
                "Windows": ".dll",
            }.get(platform.system(), ".so")
            fullname_components = fullname.split(".")
            ext_fullname = "".join(
                (
                    (
                        os.path.join(*fullname_components[:-1]),
                        os.path.sep,
                    )
                    if len(fullname_components) > 1
                    else tuple()
                )
                + (
                    shared_lib_prefix,
                    fullname_components[-1],
                    shared_lib_ext,
                )
            )
            return ext_fullname
        return super().get_ext_filename(fullname)


class CTypes(Extension):
    pass


setup(
    name="testct",
    version="1.0",
    ext_modules=[CTypes("ct", sources=["testct/ct.c"]), Extension("ext", sources=["testct/ext.c"])],
    cmdclass={"build_ext": build_ext},
)

IMHO, this is currently the best approach. It supports editable installs and wheel builds.

Answer 3

I have setup a minimal working python package with ctypes extension here: https://github.com/himbeles/ctypes-example which works on Windows, Mac, Linux.

• It takes the approach of memeplex above of overwriting build_ext.get_export_symbols() and forcing the library extension to be the same (.so) for all operating systems.
• Additionally, a compiler directive in the c / c++ source code ensures proper export of the shared library symbols in case of Windows vs. Unix.
• As a bonus, the binary wheels are automatically compiled by a GitHub Action for all operating systems :-)

Answer 4

The distutils documentation here states that:

A C extension for CPython is a shared library (e.g. a .so file on Linux, .pyd on Windows), which exports an initialization function.

So the only difference regarding a plain shared library seems to be the initialization function (besides a sensible file naming convention I don't think you have any problem with). Now, if you take a look at distutils.command.build_ext you will see it defines a get_export_symbols() method that:

Return the list of symbols that a shared extension has to export. This either uses 'ext.export_symbols' or, if it's not provided, "PyInit_" + module_name. Only relevant on Windows, where the .pyd file (DLL) must export the module "PyInit_" function.

So using it for plain shared libraries should work out-of-the-box except in Windows. But it's easy to also fix that. The return value of get_export_symbols() is passed to distutils.ccompiler.CCompiler.link(), which documentation states:

'export_symbols' is a list of symbols that the shared library will export. (This appears to be relevant only on Windows.)

So not adding the initialization function to the export symbols will do the trick. For that you just need to trivially override build_ext.get_export_symbols().

Also, you might want to simplify the module name. Here is a complete example of a build_ext subclass that can build ctypes modules as well as extension modules:

from distutils.core import setup, Extension
from distutils.command.build_ext import build_ext


class build_ext(build_ext):

    def build_extension(self, ext):
        self._ctypes = isinstance(ext, CTypes)
        return super().build_extension(ext)

    def get_export_symbols(self, ext):
        if self._ctypes:
            return ext.export_symbols
        return super().get_export_symbols(ext)

    def get_ext_filename(self, ext_name):
        if self._ctypes:
            return ext_name + '.so'
        return super().get_ext_filename(ext_name)


class CTypes(Extension): pass


setup(name='testct', version='1.0',
      ext_modules=[CTypes('ct', sources=['testct/ct.c']),
                   Extension('ext', sources=['testct/ext.c'])],
      cmdclass={'build_ext': build_ext})

Answer 5

Some clarifications here:

1. It's not a "ctypes based" library. It's just a standard C library, and you want to install it with distutils. If you use a C-extension, ctypes or cython to wrap that library is irrelevant for the question.

2. Since the library apparently isn't generic, but just contains optimizations for your application, the recommendation you link to doesn't apply to you, in your case it is probably easier to write a C-extension or to use Cython, in which case your problem is avoided.

For the actual question, you can always use your own custom distutils command, and in fact one of the discussions linked to just such a command, the OOF2 build_shlib command, that does what you want. In this case though you want to install a custom library that really isn't shared, and then I think you don't need to install it in /usr/lib/yourproject, but you can install it into the package directory in /usr/lib/python-x.x/site-packages/yourmodule, together with your python files. But I'm not 100% sure of that so you'll have to try.