CODEWITHSUNDEEP
c++linkershared-librariesdlopenundefined-symbol
OneRaynyDay
OneRaynyDay

Reputation: 3968

`dlopen`'ing a .so that contains symbols results in undefined symbols

Problem

To be concrete, I am using cblas for my application. I create a foo.so that uses cblas as a subroutine, and I dlopen('foo.so', RTLD_LAZY).

This is the flags I use to compile(redacted some -I flags):

g++-8 -std=c++17 -shared -O3 -xc++ 
    -ffast-math -fPIC -fopenmp -lcblas - -o /mnt/xxx/python/test.so

This is the error I get upon dlopen():

test.so: undefined symbol: cblas_dgemm

Here we see that cblas_dgemm is included in the .so.

root@cd872c4b85ff:/mnt/xxx/python# objdump -TC test.so | grep cblas
0000000000000000      D  *UND*  0000000000000000              cblas_ddot
0000000000000000      D  *UND*  0000000000000000              cblas_dsyrk
0000000000000000      D  *UND*  0000000000000000              cblas_dgemm
0000000000000000      D  *UND*  0000000000000000              cblas_dgemv

But in ldd it is not listed as a dependency.

root@cd872c4b85ff:/mnt/xxx/python# ldd test.so
    linux-vdso.so.1 (0x00007ffdbb3c4000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f43f23a7000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f43f2177000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f43f1f5f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f43f1b6e000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f43f17d0000)
    /lib64/ld-linux-x86-64.so.2 (0x0000562eb733a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f43f15cc000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f43f13ab000)

I did a test compile and it says that -lcblas was successfully found:

g++-8 -lcblas -Wl,--verbose | grep blas
attempt to open /usr/lib/gcc/x86_64-linux-gnu/8/libcblas.so failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/8/libcblas.a failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libcblas.so succeeded
-lcblas (/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libcblas.so)
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status

What does work

One important thing is that if I set my environment variable for LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcblas.so, the executable will run correctly. However, this is obviously not The Right Way to get the executable to work. -Wl,rpath= is also not The Right Way, since cblas will reside on different directories (i.e. on OS X it's in /usr/lib/libcblas.so, and in ubuntu it's /usr/lib/x86_64-linux-gnu/libcblas.so, and I'm guessing there will be more unpredictable locations).

What could be the reason? Am I not linking it correctly? Is the dynamic linker not finding the symbols for some reason?

Since some of the information is redacted, please let me know if there is anything that is needed and I'll add it to the information above. Thanks.

EDIT: The symbols do exist inside of libcblas.so:

root@3841c2760800:/usr/lib/x86_64-linux-gnu# nm -D libcblas.so | grep cblas_ddot
000000000000a0a0 T cblas_ddot
root@3841c2760800:/usr/lib/x86_64-linux-gnu# nm -D libcblas.so | grep cblas_dgemm
0000000000005790 T cblas_dgemm
root@3841c2760800:/usr/lib/x86_64-linux-gnu# nm -D libcblas.so | grep cblas_dgemv
00000000000077f0 T cblas_dgemv
root@3841c2760800:/usr/lib/x86_64-linux-gnu# nm -D libcblas.so | grep cblas_dsyrk
00000000000065f0 T cblas_dsyrk

EDIT:

The dependency graph, I believe, looks something like:

main --(dlopens)--> test.so 
    --(includes)--> header-only linear algebra library 
    --(calls cblas subroutine)--> /usr/lib/x86.../libcblas.so

Upvotes: 2

Views: 1453

Answers (1)

OneRaynyDay
OneRaynyDay

Reputation: 3968

So - it was because the ordering of my linkage. What I do is I stream in a C++ file (or string) into this g++ compiling process. To stream, one would do:

g++ ... -xc++ - ...

Where -xc++ is telling g++ it's taking in a c++ file(mandatory), and - is the placeholder for the file string. In my compilation, I have:

g++ ... linker_flags ...  -xc++ -

Which means that it actually wasn't getting linked correctly(linker_flags need to be after the file name).

I knew that the ordering is important, but in this situation when I'm streaming in a file, I forgot about this principle and that's why the symbols were undefined.

Upvotes: 1

Related Questions