CODEWITHSUNDEEP
pythonnamespacesfunction-declarationpython-cffi
JamesTheAwesomeDude
JamesTheAwesomeDude

Reputation: 1052

Renaming/re-declaring functions for CFFI?

I have a C library for which I'm trying to create (out-of-line, API mode) CFFI bindings. The C library provides various implementations of each function, but all of them have this giant, obnoxious prefix added onto them.

For example, the AVX-optimized implementation of the foo function from the GreenSpam submodule is named THELIBRARY_GREENSPAM_OPTIMIZED_AVX_foo(); I'll be exposing this as TheLibrary.Spam.GreenSpam.foo, with CPU optimizations chosen transparently at import-time.

I am capable of determining this prefix for each module at ffibuilder.compile()-time, so I'd like to try to lean into DRY to the highest degree possible. My code currently looks like this, which as you can see is about as horrible as it gets as regards DRY and doesn't provide any easy or obvious avenue for moving forward at the run-time algorithm choice:

# TheLibrary/Spam/GreenSpam.py
from . import _greenspam_generic
from . import _greenspam_optimized_avx  # TODO

__all__ = ['foo', 'oof']

def foo(bar):
    """GreenSpam foo"""
    _out = _greenspam_generic.ffi.new(f"uint8_t[{len(bar):d}]")
    _greenspam_generic.lib.THELIBRARY_GREENSPAM_GENERIC_foo(bar, len(bar), _out)
    # void function never errors
    return bytes(_out)

def oof(baz):
    """GreenSpam oof"""
    _out = _greenspam_generic.ffi.new(f"uint8_t[{len(baz):d}]")
    err = _greenspam_generic.lib.THELIBRARY_GREENSPAM_GENERIC_oof(baz, len(baz), _out)
    if err:
        raise RuntimeError("oof")
    return bytes(_out)

# TheLibrary/Spam/PurpleSpam.py
from . import _purplespam_generic
from . import _purplespam_optimized_avx  # TODO

__all__ = ['foo', 'oof']

def foo(bar):
    """PurpleSpam foo"""
    _out = _purplespam_generic.ffi.new(f"uint8_t[{len(bar):d}]")
    _purplespam_generic.lib.THELIBRARY_PURPLESPAM_GENERIC_foo(bar, len(bar), _out)
    # void function never errors
    return bytes(_out)

def oof(baz):
    """PurpleSpam oof"""
    _out = _purplespam_generic.ffi.new(f"uint8_t[{len(baz):d}]")
    err = _purplespam_generic.lib.THELIBRARY_PURPLESPAM_GENERIC_oof(baz, len(baz), _out)
    if err:
        raise RuntimeError("oof")
    return bytes(_out)

I would really like to flatten out these C library function names when I'm running ffi.compile(), ideally exposing aliases something like (fr'\b{re.escape(prefix)}_(\w+)\b', r'\1') to make my bindings' code cleaner and better, and in particular to enable me to add in those import-time CPU choices without having to multiplicatively duplicate my code again.

I'd like to track the upstream library as closely as possible, so doing transformations on the C source to remove these prefixes would be painful and I'd like to ask in particular about alternatives to doing that.

However, I would be fine to add, for example during ffibuilder.set_source(), normalized aliases to the mangled names, especially if that could be done in a DRY-focused way. (I did try function references, but apparently those are C++ only?)

The specific library I'm currently looking at is PQClean, but I expect I'll run up against this in future bindings as well, so that's another point against drilling down into transforming the C source too much, because I'll have to re-duplicate that effort each time I run across a new library, and possibly again each time that library refactors their header spaghetti.

Upvotes: 2

Views: 50

Answers (1)

Armin Rigo
Armin Rigo

Reputation: 12990

You can certainly play some Python-level tricks. I would do something like... (admittedly, this uses quite a bit of various Python hacks)

# to make the builds "mylib_<suffix>.so"
for suffix in ['generic', 'optimized_avx']:
    builder = FFIBuilder()
    builder.set_source("mylib_" + suffix,
    f"""
        #include <mylib.h>
        int foo_{suffix}(int x, int y);
        void bar_{suffix}(void);
    """)
    builder.compile()

# minimal source code for mylib_generic.py
import _mylib_maker
globals().update(_mylib_maker.make("generic"))

# minimal source code for mylib_optimized_avx.py
import _mylib_maker
globals().update(_mylib_maker.make("optimized_avx"))

# actual Python-side wrappers defined in _mylib_maker.py
def make(suffix):
    mod = __import__("mylib_" + suffix)
    c_foo = getattr(mod, "foo_" + suffix)
    c_bar = getattr(mod, "bar_" + suffix)

    my_funcs = []

    @my_funcs.append
    def foo(x, y):
        result = c_foo(x, y)
        if result < 0: raise Exception
        return result

    @my_funcs.append
    def bar():
        return c_bar()

    return {func.__name__: func for func in my_funcs}

Upvotes: 1

Related Questions