g++ issues with shared libraries and libpthread.so

Question

I've hit a wall with this issue and I'm hoping I can find some help here with this issue. I've created a small sample executable and shared library that exhibits the problem.

Sorry, I realize this has turned into a wall of text, but I was trying to make sure to include all the relevant information.

My Setup

System: CentOS release 5.11 (Final)
g++: gcc version 4.4.7 20120313 (Red Hat 4.4.7-1) (GCC) 
libc.so.6: Compiled by GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-55).

I have also tried this on a Redhat 6.6 machine with similar results.

My scenario:

I have an application that is trying to load a shared library at runtime via ::dlopen(). If I don't link in pthread then it appears to work but it will eventually crash in the shared library trying to throw an exception. The reason for this is that the system runtime libraries were built expecting thread local storage (TLS) and the exception processing uses a data structure from TLS but in this case it's NULL and it causes a crash. The functions are __cxa_allocate_exception and __cxa_get_globals, and it looks like they are using the stub functions from libc since pthread isn't linked in.

The problem I'm having now is trying to link in pthread to correct the issue mentioned above. If I build with pthreads, the application segfaults trying to load libpthread.so.0 as a dependency of my shared library. Everything I've read about this crash is that the the application was built without pthread while the shared library was built with pthread. However I'm building both binaries with pthreads and I still experience the issue.

Sample Code:

Shared Library Files (foo.*)

foo.h

#pragma once
extern "C"
{
    extern void DoWork();
}

foo.cpp

#include "foo.h"
#include <stdio.h>

void DoWork()
{
    printf( "SharedLibrary::DoWork()\n" );
}

Application File (main.cpp)

main.cpp

#include "foo.h"
#include <stdio.h>
#include <dlfcn.h>

void LoadSharedLibrary()
{
    void* handle = 0;
    void(*function)();

    try
    {
        printf( "Loading the shared library\n" );
        handle = ::dlopen( "libfoo.so", 2 );
        function = (void (*)())::dlsym( handle, "DoWork" );
        printf( "Done loading the shared library\n" );

        function();
    }
    catch(...)
    {
        printf( "ERROR - Exception while trying to load the shared library\n" );
    }
}

int main(int argc, char* argv[])
{
    LoadSharedLibrary();
    return 0;
}

Explicit Loading

Trying to load the shared library at runtime using the following build script results in a segfault trying to load libpthread.so.0.

Build script:

compiler=g++
arch=-m32
echo gcc architecture flag: ${arch}

${compiler} -c -fPIC -g ${arch} -pthread -o ./foo.o foo.cpp
${compiler} ${arch} -shared -g -o ./libfoo.so ./foo.o -lpthread

${compiler} -c -fPIC -g ${arch} -pthread -o ./main.o main.cpp
${compiler} ${arch} -static -g -o main.out ./main.o -lpthread -ldl -lc 

The stack trace for this crash is:

#0  0x00000000 in ?? ()
#1  0x0089a70a in __pthread_initialize_minimal_internal () at init.c:417
#2  0x0089a218 in call_initialize_minimal () from /lib/libpthread.so.0
#3  0x00899da8 in _init () from /lib/libpthread.so.0
#4  0x0808909b in call_init ()
#5  0x080891b0 in _dl_init ()
#6  0x08063a87 in dl_open_worker ()
#7  0x0806245a in _dl_catch_error ()
#8  0x0806349e in _dl_open ()
#9  0x08053106 in dlopen_doit ()
#10 0x0806245a in _dl_catch_error ()
#11 0x08053541 in _dlerror_run ()
#12 0x08053075 in __dlopen ()
#13 0x0804830f in dlopen ()
#14 0x0804824f in LoadSharedLibrary () at main.cpp:13
#15 0x080482d3 in main (argc=1, argv=0xffffd3e4) at main.cpp:27

The loaded shared libraries are:

From        To          Syms Read   Shared Object Library
0xf7ffb3b0  0xf7ffb508  Yes         libfoo.so
0x0089a210  0x008a5bc4  Yes (*)     /lib/libpthread.so.0
0xf7f43670  0xf7fbec24  Yes (*)     /usr/lib/libstdc++.so.6
0x009a8410  0x009c35a4  Yes (*)     /lib/libm.so.6
0xf7efb660  0xf7f02f34  Yes (*)     /lib/libgcc_s.so.1
0x0074dcc0  0x0084caa0  Yes (*)     /lib/libc.so.6
0x007197f0  0x0072f12f  Yes (*)     /lib/ld-linux.so.2
(*): Shared library is missing debugging information.

Implicit Loading

This uses a different build script that tries to setup the dependency at build time and would in theory not require an explicit load call. This isn't a valid use case for our real world scenario, but I tried to do this while looking into this issue.

Build Script:

compiler=g++
arch=-m32
echo gcc architecture flag: ${arch}

${compiler} -c -fPIC -g ${arch} -pthread -o ./foo.o foo.cpp
${compiler} ${arch} -shared -g -o ./libfoo.so ./foo.o -lpthread

${compiler} -c -fPIC -g ${arch} -pthread -o ./main.o main.cpp
${compiler} ${arch} -static -g -L. -o main.out ./main.o -lpthread -ldl -Wl,-Bdynamic -lfoo -Wl,-static -lc

Behavior:

Starting program: /app_local/dev3/stack_overflow/main.out 
/bin/bash: /app_local/dev3/stack_overflow/main.out: /usr/lib/libc.so.1: bad ELF interpreter: No such file or directory
/bin/bash: /app_local/dev3/stack_overflow/main.out: Success

During startup program exited with code 1.

The weird thing is that I've done objdump -p <library> | grep NEEDED and none of the libraries in the dependency chain have libc.so.1 as a dependency. The version of libc they depend on is libc.so.6.

End of build scenarios

I'm really hoping somebody here has an idea about what is going on and could help me out. My Google and StackOverflow skills have failed me as everything I've found points to mismatched pthread usage as the root problem.

Thanks in advance!

Answer 1

${compiler} ${arch} -static -g -o main.out ./main.o -lpthread -ldl -lc

This is a fully-static link.

On most OSes, no calls to dlopen can be made from a fully-static binary (dlopen is simply not provided in libdl.a, and the link fails).

GLIBC is an exception, but only so far as dlopen is needed to support /etc/nsswitch.conf. Almost certainly dynamic loading of libpthread.so.0 into a fully-static a.out that contains its own copy of libpthread.a is not supported. The short answer is: it hurts, don't do that.

Fully-static linking is in general a very bad idea on any modern UNIX system. Fully-static linking of multithreaded apps doubly so. Fully-static linking that then dynamically loads another copy of libpthread? Really bad idea.

Update:

GLIBC consists of many libraries (200+), and I would strongly advise against mixing static and dynamic linking for any such library. In other words, if you link against libc.a, then make it a completely static link. If you link against libc.so, then don't statically link libpthread.a, libdl.a, or any other part of GLIBC.