CODEWITHSUNDEEP
pythonmacosclanghomebrewdistutils
Andrew Collette
Andrew Collette

Reputation: 751

Homebrew/Python: Convince distutils to link against a specific library on OS X?

I'm building a Python project with multiple C extensions which require libhdf5. I have an existing installation of libhdf5 at /usr/local/lib. For testing and development, I want to develop against a private build of HDF5 located at /Users/name/some/path.

In setup.py, I handle this by setting the "library_dirs" (and "runtime_library_dirs", although on OS X that doesn't do anything) to /Users/name/some/path. On Linux this works fine.

The trouble is, when my extension modules are compiled, they link against the copy of HDF5 in /usr/local/lib and no amount of tweaking in setup.py can convince them otherwise.

By setting DYLD_LIBRARY_PATH=/Users/name/some/path when running Python, I have successfully loaded the private build of HDF5, so I know the library is built correctly and works.

Running otool -L on one of my extensions produces this:

h5py/_errors.so:
/usr/local/lib/libhdf5.8.dylib (compatibility version 9.0.0, current version 9.1.0)
/usr/local/lib/libhdf5_hl.8.dylib (compatibility version 9.0.0, current version 9.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

which confirms that we're linking against the wrong copy of the library. I noticed that the linking phase looks like this:

clang -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib -L/Users/name/some/path <more stuff>

and conjecture that clang is linking against the first version of HDF5 it can find.

Is there any way to force distutils to link against my private build of the library? I don't need a relocatable version so I don't care about @rpath, etc.

I have also confirmed that this doesn't happen with the system Python, just the one installed via homebrew.

Upvotes: 4

Views: 794

Answers (1)

Aron Ahmadia
Aron Ahmadia

Reputation: 2405

My short answer is: Don't use the HomeBrew Python to manage builds outside of homebrew. You can see here: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python.rb#L213 that they've hard-coded their library flags into the Python Makefile itself.

Of course, an answer that fixes your problems is much more interesting :)

You could, if you really wanted to, monkey-patch sysconfig.get_config_vars() to inject your flags ahead of system libraries. You could also monkey-patch distutils. Neither of these is particularly robust.

Within HashDist we've prioritized building our own Python, which is frustrating, but we've found to be the most robust. (We support h5py on OS X)

Upvotes: 1

Related Questions