any way to set cython compiling with ucs2?

Question

Hit problem when convert python code to shared object by Cython.

setup file here:

from distutils.core import setup
from Cython.Build import cythonize
setup(
    ext_modules = cythonize("hello.py")
)

So everything works fine on my Ubuntu desktop util transferred to CentOS.

Got error:

undefined symbol: PyUnicodeUCS4_DecodeUTF8

I googled and find there are many questions on this, but, almost all of them say the root cause is python with UCS2 or UCS4, and I understand this, didn't find one show the way to solve this.

IMO, ways to solve:

1. rebuild python to get the right version by "--enable-unicode=ucs4/ucs2"

But I need to reinstall all packages

2. Compile the code from another desktop whose python with the right UCS

Now, I wanna if there is way to set Cython to compile with specified UCS mode.

Any suggestions is great appreciated.

Thanks.

Answer 1

First, to answer your actual question:

I wanna if there is way to set Cython to compile with specified UCS mode.

You can build a separate python installation from source and link Cython against its headers. To find the headers, you can use the python-config tool (or python3-config for Python 3). It is usually located in the bin directory where the python executable is:

$ # system python on my machine (macos):
$ which python-config
/usr/bin/python-config

$ # python 3 installation
$ which python3-config 
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3-config

$ python-config --cflags
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE
$ python-config --ldflags
-L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl -framework CoreFoundation

Copy the output to the setup.py:

from setuptools import setup
from setuptools.extension import Extension
from Cython.Build import cythonize


cflags_ucs4 = [
    '-I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m',
    '-I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m',
    ...
]
ldflags_ucs4 = [
    '-L/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin', 
    '-lpython3.6m', 
    ...
]

cflags_ucs4 = [...]
ldflags_ucs2 = [...]


should_build_ucs2 = False  # i.e. could be passed via sys.argv

if should_build_ucs2:
    cflags = cflags_ucs2
    ldflags = ldflags_ucs2
else:
    cflags = cflags_ucs4
    ldflags = ldflags_ucs4

extensions = [
    Extension('hello.py', extra_compile_args=cflags, extra_link_args=ldflags),
]

setup(
    ext_modules = cythonize(extensions)
)

---

However, I do not recommend doing that as you won't win anything by doing that - you will still need to build and distribute two separate packages (one for UCS2, another for UCS4) which is messy to maintain.

Instead, if you are building a wheel that should be installable on a wide range of Linux distros (what is most probably your actual goal), I would suggest to make your build compliable with PEP 513 (manylinux1 packages).I suggest you to read it through as it was very helpful for me when I faced the problem of distributing Linux-compliant wheels.

Now, one way to get a manylinux1-compliant wheel is to build the wheel on your machine, then running auditwheel to check for platform-specific issues and trying to resolve them:

$ pip install auditwheel
$ python setup.py bdist_wheel
$ # there should be now a mypkg-myver-cp36-cp36m-linux_x86_64.whl file in your dist directory
$ auditwheel show dist/mypkg-myver-cp36-cp36m-linux_x86_64.whl
$ # check what warnings auditwheel produced
$ # if there are warnings, try to repair them:
$ auditwheel repair dist/mypkg-myver-cp36-cp36m-linux_x86_64.whl

This should generate a wheel file named mypkg-myver-cp36-cp36m-manylinux1_x86_64.whl in a wheelhouse directory. Check again that everything is fine now by running auditwheel show wheelhouse/mypkg-myver-cp36-cp36m-manylinux1_x86_64.whl. If the wheel is now consistent with manylinux1, you can distribute it and it should work on most Linux distros (at least those with glibc; distros with musl like Alpine won't work, you will need to build a separate wheel if you want to support it).

What should you do if auditwheel can't repair your wheel? The best way is to pull a special docker container provided by PyPA for building manylinux1-compliant wheels (this is what I'm using myself):

$ docker pull https://quay.io/repository/pypa/manylinux1_x86_64

A wheel built inside this container will work on most of the Linux distros (excluding some exotic ones like Alpine).