Cross compiling with CMake: linker errors

Question

I am attempting to cross compile a project for my raspberry pi (32 bit armv8) on my ubuntu machine. I set up a toolchain using crosstool-NG and compiling works, but linking fails. My CMake toolchain file is as follows:

SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)

set(CMAKE_SYSROOT /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/sysroot)

SET(CMAKE_C_COMPILER /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/bin/armv8-rpi3-linux-gnueabihf-gcc)
SET(CMAKE_CXX_COMPILER /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/bin/armv8-rpi3-linux-gnueabihf-g++)

SET(CMAKE_FIND_ROOT_PATH /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf)

SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
# for libraries and headers in the target directories
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

I have a few libraries that are not being found by the linker. The problem is with my sysroot. I copied over /usr and /lib from the pi to my sysroot folder that was created by crosstool-NG. The libraries that are being linked reside in sysroot/usr/lib/arm-linux-gnueabihf.

However, the linker is not checking that directory, it checks sysroot/usr/lib and a few others, but for some reason it does not check the arm-linux-gnueabihf directory.

On the pi, I can run ld -lasound --verbose and get the following output:

attempt to open //usr/local/lib/arm-linux-gnueabihf/libasound.so failed
attempt to open //usr/local/lib/arm-linux-gnueabihf/libasound.a failed
attempt to open //lib/arm-linux-gnueabihf/libasound.so failed
attempt to open //lib/arm-linux-gnueabihf/libasound.a failed
attempt to open //usr/lib/arm-linux-gnueabihf/libasound.so succeeded

so its finding it correctly on the pi.

When I do the same using the ld generated by crosstool-NG:

attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/home/stone/x-tools/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/lib/libasound.so failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/home/stone/x-tools/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/lib/libasound.a failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/usr/local/lib/libasound.so failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/usr/local/lib/libasound.a failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/lib/libasound.so failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/lib/libasound.a failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/usr/lib/libasound.so failed

It fails to find it because it doesnt look in the arm-linux-gnueabihf directory. I copied the library .so file from sysroot/usr/lib/arm-linux-gnueabihf to sysroot/usr/lib and it compiles and links successfully, but I would like to make it so that I do not have to do that. How can I make the linker check that arm-linux-gnueabihf directory?

EDIT: I also copied /etc/ld.so.conf and /etc/ld.so.conf.d from the pi into my sysroot, but it does not seem to affect the linker.

EDIT: After further research it looks like this might be due to something with gcc-multiarch. I am not sure what that is but hopefully I can figure this out

EDIT: I verified that indeed the linker is not searching the arm-linux-gnueabihf path:

 ./armv8-rpi3-linux-gnueabihf-ld --verbose | grep -i "search"

outputs:

SEARCH_DIR("=/home/stone/x-tools/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/lib"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib");

Answer 1

Alright I got it all figured out. The repo where I learned the tricks to getting this to work can be found here. I am grateful for this repo, without it I would not have gotten this to work.

The issue lies with the linker. The linker that comes with plain old binutils that gets pulled in by crosstool-ng does not search the arm-linux-gnueabihf subdirectories. These directories exist because of Debian multiarch, and in order to get a linker that looks in such a directory, binutils needs to be patched. The instructions that follow assume you have already generated a ct-ng config based on the rpi3 sample.

On the raspberry pi, I installed the binutils-source package:

$ sudo apt install binutils-source

This will give access to a patch file located in /usr/src/binutils/patches.

On the host machine, you need to add a patches directory in the directory where you are building your toolchain (where the ct-ng config file is). It must have a specific structure that reflects the structure of packages in the crosstool-ng repo (ie. inside patches/ you must have a directory with the package name, inside that you must have the version number, and inside that is where you place the patch.):

$ cd your_toolchain_directory
$ mkdir -p patches/binutils/2.31.1/

Now we can copy the patch file from the pi to the host. The specific patch we need is 129_multiarch_libpath.patch:

$ cd patches/binutils/2.31.1/
$ scp pi@raspberrypi:/usr/src/binutils/129_multiarch_libpath.patch

Now that we have the patch, we need to update the config file to tell ct-ng to include local patches as well as enable the multiarch flag for gcc. You can do this using ct-ng menuconfig so change back to the directory with the config file and run:

$ ct-ng menuconfig

In the paths and misc options you need to change the patches origin to bundled then local, and then add ${CT_TOP_DIR}/patches as the local patch directory. This should produce the following lines under the extracting section of the .config file:

CT_PATCH_BUNDLED_LOCAL=y
CT_PATCH_ORDER="bundled,local"
CT_PATCH_USE_LOCAL=y
CT_LOCAL_PATCH_DIR="${CT_TOP_DIR}/patches"

Next you need to add the --enable-multiarch flag in the gcc options. Again using menuconfig, go to the C compiler settings. Add --enable-multiarch to the gcc extra config setting. The config file should have the following in the gcc settings:

CT_CC_GCC_EXTRA_CONFIG_ARRAY="--enable-multiarch"

Save the config. The last thing we need to do is export a variable.

$ export DEB_TARGET_MULTIARCH=arm-linux-gnueabihf

Now we can build the toolchain:

$ ct-ng build

Once its done, you can change into the bin directory of the generated toolchain and run:

$ ./ld --verbose | grep -i "search"

and you should see that the arm-linux-gnueabihf directory is now in the search paths. What a process.

Answer 2

You need to use static library instead. then the executable will include the binary of the libraries. But keep in mind the executable file will be bigger ( depends obviously which lib...). When you using cross compile, you should check that the libraries you use exist in the other environment you execute the program. The solution will be to add STATIC word according to CmakeList.txt file