Dependency Hell: linux .so plugin dynamic loading

Question

I've created a shared library built with a standalone build tree using linuxbrew, which fails to load in the parent application due to dependency conflict. I'm using a separate application which dynamically loads the library after startup with the Qt5 QLibrary class.

My library is libv_repExtPluginSkeleton.so. Both it and the parent app depend on glibc and libstdc++. All of the main app's dependencies are in /usr/lib, while all my library's dependencies are in ~/.linuxbrew/lib.

When the parent application goes to load the .so failed I debugged the failure with LD_DEBUG=all "$dirname/$appname" and found the following error report in the output:

  2610: file=/home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0];  dynamically loaded by libQt5Core.so.5 [0]
  2610: file=/home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0];  generating link map
  2610:   dynamic: 0x00007fd063cff570  base: 0x00007fd063ae7000   size: 0x000000000021a6a8
  2610:     entry: 0x00007fd063af1150  phdr: 0x00007fd063ae7040  phnum:                  5
  2610: 
  2610: checking for version `GCC_3.0' in file /lib/x86_64-linux-gnu/libgcc_s.so.1 [0] required by file /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0]
  2610: checking for version `GLIBC_2.14' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0]
  2610: checking for version `GLIBC_2.2.5' in file /lib/x86_64-linux-gnu/libc.so.6 [0] required by file /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0]
  2610: checking for version `CXXABI_1.3' in file /usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0] required by file /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0]
  2610: checking for version `GLIBCXX_3.4.9' in file /usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0] required by file /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0]
  2610: checking for version `GLIBCXX_3.4.21' in file /usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0] required by file /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0]
  2610: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: error: version lookup error: version `GLIBCXX_3.4.21' not found (required by /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so) (fatal)
  2610: 
  2610: file=/home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so [0];  destroying link map

As you can see, it seems that the glibc version already loaded is in /usr/lib when my library loads, but my library needs to be loading the version in ~/.linuxbrew/lib.

QLibrary is used to load the plugin as follows:

plug=new CPlugin(filename,pluginName);
int loadRes=plug->load();

I read in The Inside Story on Shared Libraries and Dynamic Loading that "dynamically loaded modules are completely decoupled from the underlying application", and if that can be made to work correctly I'd like to reconfigure the loading process to solve the problem.

Here are the dependency lists found by ldd, which illustrates the dependency overlaps in different locations:

+ ldd /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/libv_repExtPluginSkeleton.so
    linux-vdso.so.1 (0x00007fffc17af000)
    libstdc++.so.6 => /home/hbr/.linuxbrew/lib/libstdc++.so.6 (0x00007ff5b9a32000)
    libm.so.6 => /home/hbr/.linuxbrew/lib/libm.so.6 (0x00007ff5b9742000)
    libgcc_s.so.1 => /home/hbr/.linuxbrew/lib/libgcc_s.so.1 (0x00007ff5b9531000)
    libc.so.6 => /home/hbr/.linuxbrew/lib/libc.so.6 (0x00007ff5b91b9000)
    /home/hbr/.linuxbrew/Cellar/glibc/2.19/lib64/ld-linux-x86-64.so.2 (0x00007ff5b9f81000)
+ ldd /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/vrep
    linux-vdso.so.1 =>  (0x00007ffc333f9000)
    liblua5.1.so (0x00007fc10e763000)
    libQt5Core.so.5 (0x00007fc10df28000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc10dd0a000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc10da06000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc10d7f0000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc10d42b000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc10d125000)
    libicui18n.so.54 (0x00007fc10ccb7000)
    libicuuc.so.54 (0x00007fc10c909000)
    libicudata.so.54 (0x00007fc10aedf000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc10acdb000)
    libgthread-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0 (0x00007fc10aad9000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc10a8d1000)
    libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fc10a5c9000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fc10e66d000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fc10a38b000)
+ LD_DEBUG=all /home/hbr/V-REP_PRO_EDU_V3_2_2_64_Linux/vrep

My launch script sets the first entry in LD_LIBRARY_PATH to be ~/.linuxbrew/lib but the application still picks up the system version first.

Also, for my own plugin library the RPATH is correct:

objdump -x libv_repExtPluginSkeleton.so |grep RPATH
RPATH                /home/hbr/.linuxbrew/lib

How can I resolve this version conflict so that my library loads correctly?

Also, is this possible without changing the complete toolchain of either the parent application or my libary?

Answer 1

This answer will be specific to my use case, but many of the commands and settings can be generalized to other applications and configurations. I also have a more detailed discussion of my debugging of this specific problem on the vrep forum.

First I changed vrep.sh to completely clear LD_LIBRARY_PATH, so it doesn't interfere with what's loading.

unset LD_LIBRARY_PATH

Thanks to @fireant for the pointer, though unfortunately that link alone didn't solve my problem.

My first hints towards solving the problem were from this stackoverflow post on rpath following commands from that post that let you see the current rpath:

objdump -x binary-or-library |grep RPATH

# Maybe an even better way to do it is the following:
readelf -d binary-or-library |head -20

# This second command above also lists the direct dependencies on other libraries followed by rpath.

# On ubuntu 15.04 someone had to use:
objdump -x binary-or-library |grep RUNPATH

The current 3.2.2 release of vrep has a very strange rpath, which I believe is because it may not be explicitly configured when vrep is built for release:

$ objdump -x vrep |grep RPATH
  RPATH                /home/marc/Qt5.2.0/5.2.0/gcc_64:/home/marc/Qt5.2.0/5.2.0/

To solve the problem in my case I created a symlink from $VREPDIR/lib to where I wanted my libraries to be found in ~/.linuxbrew/lib, then I changed the rpath of the vrep executable with the following commands:

VREPDIR=/path/to/vrep/executable
cd $VREPDIR
ln -s ~/.linuxbrew/lib $VREPDIR/lib
patchelf --set-rpath '$ORIGIN/lib' vrep

Note that the ld/rpath/patchelf systems understand $ORIGIN to be the directory where the vrep executable is located. Once I've run the above code, I have a new RUNPATH as follows (RPATH vs runpath may be due to version differences?):

$ objdump -x vrep |grep RUNPATH
  RUNPATH              $ORIGIN/lib

This solution isn't ideal and I'll probably need to fix it up because it is picking up the linuxbrew Qt installation as follows:

libQt5Core.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007fece8993000)

This may conflict with the API used by V-REP in the Qt shared library provided in the directory alongside vrep, especially as linuxbrew is updated. Nonetheless this solution works and vrep is loading 2 out of 3 .so libraries without crashing! One improvement will be a few minor changes like those outlined below so that the libraries provided with vrep are still loaded.

Actually, the ideal solution for me would be one where vrep loads the system libraries, and the plugin loads the .linuxbrew libraries. I'm not sure that is possible and if it is it may require modifications to the vrep source/build.

While the application works with this solution there is some potential problems and I'm not sure if it is the ideal solution. Nonetheless I think it may be useful to others who encounter a similar issue with the applications they use.

Answer 2

You should have a config file somewhere on your file system where the default modules folder is defined for your application. Vim and edit this path to be ~/.linuxbrew/lib. Re-run. If you still get module errors, re-install the modules of the correct version as your error messages show and copy the modules to the ~/.linuxbrew/lib path. Lastly, always examine permissions & ownership on the modules and/or the module path. Perhaps your application is unable to pick up these modules due to a simple ownership/permissions issue.

Answer 3

As far as I understand, you want to build a shared library portable across different Linux distributions. This is tricky. Especially since you're dealing with C++.

Essentially, there are 3 libraries you need to deal with: libc, libgcc_s, and libstdc++ (libm is not a huge deal, since you can always statically link against it).

The first one (libc) seems to be the biggest problem, since you can neither statically link against it, nor ship with your own version. But you don't need to if you pull a simple trick. The GNU C Library uses versioning of symbols and is backward compatible. You can rely on this. Just take a reasonably old version of libc.so.6 (version 2.11 is old enough to make sure you'll work on virtually all distributions that are in use today, but you may want a newer version) and link dynamically against it. Don't ship it with your libv_repExtPluginSkeleton.so, though. You're linking against it to make sure your libv_repExtPluginSkeleton.so will dynamically link against (potentially newer) system-provided versions on target machines. You are relying on backward compatibility.

Now, if you were building an executable, you could just dynamically link against your own (rpathed) versions of libgcc_s and libstdc++, and you would be ready to go. However, since you're building a shared library, you don't have such luxury. So you have two options.

First, you can try to pull the same trick as I've advised doing with libc. Dynamically link against an older version and see if it works. In theory, this should work, since libstdc++ uses ELF symbols versioning as well (and sort of maintains backward compatibility). But I've tried doing similar things and faced several particularly nasty problems. So test rigorously (try throwing exceptions from your libv_repExtPluginSkeleton.so, try passing short and empty std::strings around, try everything).

Alternatively, you can statically link against libgcc_s and libstdc++. This is what I recommend. This could be easily done with -static-libgcc and -static-libstdc++ options. But there is, of course, a catch. It is no longer safe for your libv_repExtPluginSkeleton.so to provide a C++ interface or call C++ functions from other libraries. The reason to this is that libstdc++ is not particularly compatible between different versions.

So, summing all previously mentioned up, this is my recommendation:

• Dynamically link against an older version of libc to make sure you'll link against newer versions as well. Don't ship with it, though.
• Statically link against libgcc_s and libstdc++ to not depend on them.
• Provide a C interface, since you have no other (portable) options.

Of course, this violates your wish not to change the toolchain you're using to build your library. You lose the C++ interface. And you also have to trade libc features for compatibility. But at least you get compatibility.

And yes, don't forget about licensing issues.

Answer 4

How can I resolve this version conflict so that my library loads correctly?

That is not going to work for the basic, low-level libraries like the libc or libstdc++. The libraries manage number of global objects (memory management, threads, environment, locales, etc) and having several different versions of them in the same address space is a recipe for disaster. What's more, the STL classes of different libstdc++ versions are not guaranteed to have the same binary layout in the memory. The std::string inside the application might not be the same as the std::string inside the library. Passing one from/to plug-in might cause a hard-to-debug crash. (The error message you get for the libstdc++ is in part a safeguard against the situation.) That leads to the problem that (even if you link the libstdc++ statically) the functions of your plug-in will not be able to accept or return STL classes as arguments/return values - only the basic C types.

Additionally, with dynamic loading, another hurdle is that Linux/UNIX linker uses a flat linear list of libraries with flat linear list of its symbols. And the libraries are identified by the internal soname, not the file name. The linker would bind the sought symbol to the first one, found by linear search through the list of libraries and their symbols. Libraries loaded at later time, would get most of their symbols satisfied from the libraries already loaded for the application itself. The libraries referenced by your plug-in might not even be looked at.

Overall, your choices are very limited:

• Static linking. Link the plug-in with static variants of the required libraries. But that does not work if you need to override the libc or libstdc++.

• Server/client approach. The plug-in library is a thin wrapper which only starts the server process, and then redirects all calls from application to plug-in via some RPC mechanism to this server application. This is the only reliable method known to me of resolving the library version conflicts. Since unlike a library, the application has more control over the dynamic linker (rpath, or in worst case with the LD_PRELOAD).

Generally, as was said by others before, you should always use the libc and libstdc++ installed with the OS, releasing different builds for different OS variants if needed.

Answer 5

Most likely rpath can help you in this situation. See this answer, for an example.