CODEWITHSUNDEEP
pythonc++linkercython
Rboreal_Frippery
Rboreal_Frippery

Reputation: 2318

Specify alternative linker when building python / cython extensions?

In an attempt to speed up the development process, I have been looking to try using either the 'gold' linker or the multithreaded 'lld' linker in place of the GNU linker when compiling some cython extensions. In my setup.py I have tried something like the following:

lld_linker = shutil.which('ld.lld')
if lld_linker:
    print("Using LLD Linker in place of GNU LD")
    os.environ['LDSHARED'] = lld_linker

However, this causes the linking process to fail with a large number of "/usr/bin/ld: ..... undefined reference to ....." errors. (Build works fine without adding this LDSHARED envvar). The failure behavior is the same whether using this internal os.environ, or exporting the envvar before calling setup.py. I have a hunch that perhaps Cython's multiprocessing approach to distributing compile jobs is not keeping the environment variable everywhere, leading to this mix of linkers?

How can I properly specify the linker, so that the setting sticks and builds the same as with the GNU ld linker?

There is a related question here: How do I specify the linker when building python extensions? ; however, it did not solve my issue, as mentioned earlier.

Upvotes: 2

Views: 421

Answers (1)

ead
ead

Reputation: 34337

Usually distutils/setuptools don't use the linker directly, but call a frontend like gcc for c-extensions or g++ for c++-extensions.

These frontends collect all necessary information - like which libraries should be passed to the linker, e.g. libstdc++ for c++-extensions, - and call the linker with right command line options. One can see it when passing -v-option to the gcc- of g++-frontend for example via extra_link_args in setup.py.

So if you force distutuls/setuptools.py to use ld directly you also should provide all the options collected by the frontend in extra_link_args, otherwise some libraries will be missing and and compilation will fail, as you see.

It's quite difficult to let the setup.py to choose another linker, yet there are some cheap options to do it locally:

  1. The default linker (e.g. /usr/bin/ld) is just a symlink, let it point to the linker of your choice.
  2. Passing -B-option via extra_link_args, i.e. -B/path/to/folder/with/my/linker. The subtle details are: 1) the linker should be called ld (create a symlink if needed) 2) some distribution (e.g. Anaconda) allready provide a -B-option, which takes predecence over the path passed via extra_link_args. In this case one possible solution would be to modify emitted command line similar as described in this SO-post.
  3. set the linker directly via LDSHARED environment variable and provide all needed options in extra_link_args.

Option 2 is probably easiest to tweak to work on any system:

  1. detect if ld.lld is presend, if yes:
  2. create a temporary folder with a symlink to it (called ld)
  3. add this temp-folder as -B-options to extra_link_args
  4. detect whether the distribution already uses -B option, manipulate command line (as described here for example) to ensure the precedence of the temp-folder.

Upvotes: 1

Related Questions