CODEWITHSUNDEEP
pythoncython
wirrbel
wirrbel

Reputation: 3299

Use Cython for exposing statically linked c-library symbols

I have a question on best practice on how to use code from a local C library with Cython. In order to avoid setting the LD_LIBRARY_PATH (or installing the C library in /usr/lib or similar directories), I think static linking would be a suitable solution for my use case.

However, interfacing the c functions that have been statically linked is non-trivial. The only way I could cimport another cython module that has statically linked C code in it was to paraphrase all exported functions with Cython function pointers (see https://github.com/HolgerPeters/cython-example/blob/master/cython-project/c_with_ptrs.pyx and https://github.com/HolgerPeters/cython-example/blob/master/cython-project/c_with_ptrs.pxd)

Just declaring the original C function in the pxd as extern does not seem to work. I have exemplary project on Github which isolates different approaches and their usage.

https://github.com/HolgerPeters/cython-example

Any ideas on this? Can I tell Cython to explicitly use the externed declarations?

Specific Stack-Trace

As the question started as a rather open one, I would like to make it more specific to encourage answers.

In the named github project I have assembled several ways of linking. I will focus now on the way that seems natural from the documentation.

https://github.com/HolgerPeters/cython-example/blob/master/cython-project/c_from_with_direct_compilation.pxd

cdef extern from "foo.h":
    extern int clib_return_3(int)

and a use case in the corresponding https://github.com/HolgerPeters/cython-example/blob/master/cython-project/c_from_with_direct_compilation.pyx

def useit():
    print(clib_return_3(4))

Invocation of c_from_with_direct_compilation.useit() works (test.sh output below).

But then when I try to use clib_return_3 from another cython file, it fails! Usage is

https://github.com/HolgerPeters/cython-example/blob/master/cython-project/c_from_with_direct_compilation_user.pyx

cimport c_from

def useit():
    print(c_from.clib_return_3(4))

bash test.sh tests all usage cases, and as you can see, the cython so where the clib_return_3 symbol is linked into can indeed use that symbol, whereas the other cython so trying to import that symbol fails at this.

======================== c_from_w/_direct_comp, interfacing fails (why?)  
Invoke from statically linked c_from_with_direct_compilation, invocation by c_from_with_direct_compilation_user
Traceback (most recent call last):
  File "", line 1, in 
ImportError: ./c_from_with_direct_compilation_user.so: undefined symbol: clib_return_3
FAILURE
======================== c_from_w/_direct_comp  
Invoke from statically linked c_from_with_direct_compilation
3
SUCCESS

Upvotes: 0

Views: 896

Answers (2)

wirrbel
wirrbel

Reputation: 3299

After staring long enough at this problem I think I know what is going on:

Cython-cdef-functions are name mangled, whereas extern functions are not name-mangled and keep their original names. Name mangling is necessary for discoverability of functions across different modules. So non-mangled names cannot be shared via cimport - for those, the normal C shared library mechanisms apply.

Upvotes: 0

amwinter
amwinter

Reputation: 3139

You can do either of:

  1. cdef extern from "path/header.h": and then list the function defs you'll use in cython. (http://docs.cython.org/src/userguide/external_C_code.html#referencing-c-header-files)
  2. declare the C functions as extern. (http://docs.cython.org/src/userguide/external_C_code.html#external-declarations)

(Both methods work in pyx and pxd files).

Side note: in your setup.py, you're using 2 different ways to include the C code (using the dynamic library & using the object file). Unless your C make rules are really complicated, a third option is to just list the .c files in the same array as the .pyx files and let distutils build them for you.

Upvotes: 1

Related Questions