CODEWITHSUNDEEP
c++linuxpluginsdynamic-loading
Martin Hierholzer
Martin Hierholzer

Reputation: 930

dlmopen and C++ libraries

(This is presumably a fairly advanced problem, sorry about this :-))

I have the problem that I need to load a plugin (a shared library) into an application, but the plugin could use a library which is binary incompatible to the version of the library used by the application. My idea was to use dlmopen() and load the plugin into its own namespace. I expect to get two separate copies of the binary incompatible library (and for any other common dependency even if binary compatible).

This seems to work up to a certain extend, but under certain circumstances I get a segfault deep inside glibc, at the point where the constructors of static objects are called (this is what I found out with the debugger).

I have made a minimal example to reproduce the issue, which can be found on github: https://github.com/mhier/segregatedLinkingExample

The example uses libxml++ as an external, common C++ library, so you will need its development package to be installed. Run "mk.sh" to compile and then "main". It will then crash (at least it does on Ubuntu 16.04 and 18.04). If you remove the "-DWITH_CRASH" it no longer crashes.

The WITH_CRASH compile switch enables the use if libxml++ inside the main executable. It is always used in the plugin library libC. Only of libxml++ is used in both the main executable and the plugin I see the crash. "Using" in this context is as little as deriving a virtual class from it and making sure code for the derived class really gets generated by implementing the constructor/destructor. It is not even executing code in the plugin (other than via dl_init -> constructors of static objects etc.).

I cannot find much on the Internet about dlmopen. I have not found any bug reports pointing in the right direction. Has anyone ever used dlmopen with a new namespace for C++ libraries? Any form of input how to continue from this point is very welcome!

Upvotes: 3

Views: 1761

Answers (2)

Reimar
Reimar

Reputation: 56

The problem is not related to C++.

It is a bug in the glibc version of libpthreads, which results in libraries loaded with dlmopen returning duplicates for pthread_key_create, resulting in thread-specific storage being clobbered (same key means same memory location, it's like malloc returning the same memory area multiple times).

The reason this crashes immediately is because libglib makes heavy use of thread-specific storage already in its on-load functions.

In detail, the problem is the direct use of __pthread_keys global variable which should instead be loaded via the thread descriptor (THREAD_SELF), thus ensuring thread-local keys are allocated in a structure shared by all instances of libpthread.

See the source for details: https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/pthread_key_create.c;h=a584db412b7b550fa7f59e445155dbfddaeb1d23;hb=HEAD

Reported to glibc: https://sourceware.org/bugzilla/show_bug.cgi?id=26955

Also when debugging this kind of thing in gdb, tip to get debug symbols:

  • check /proc/$pid/maps to find out where dlmopen loaded the library
  • find the entry point of the library (e.g. readelf -h /usr/lib/x86_64-linux-gnu/libglib-2.0.so )
  • in gdb use add-symbol-file to load the symbols.
    • as file name just specify the library file, if you have debug symbols installed gdb will find them the normal way - don't try to specify the symbol file directly
    • the address is the load address from /proc/$pid/maps + entry point address

Upvotes: 4

Martin Hierholzer
Martin Hierholzer

Reputation: 930

So it seems the answer is not to do it. dlmopen seems to have issues with C++ which can result in undefined behaviour. Presumably the ODR violations are not perfectly fixed by the namespaces.

I admit, this answer is my subjective view. I have not found many good resources about using dlmopen for C++ libraries. Hence my conclusion is not to use it, as I need it to work reliably. I have seen very strange effects, e.g. my example in the question works again if I link the shared library against a particular third-party library (even without using it). Unless I can understand these effects, I would not trust a solution (as it could just work accidentally).

dlmopen() might work in other contexts, e.g. if one controls both the application and the shared library and can test if it loads properly.

Upvotes: 0

Related Questions