Linking Error on GCC and Clang: What is '__tls_get_addr@@GLIBC_2.3'

Question

I have trouble compiling the code on Ubuntu 12.04 and Ubuntu 14.04. The code is rely on libcurl, libjsoncpp and legacy version of mongo-cxx-client. It works great on Mac OS. The code is bug free on Xcode. However, it couldn't compile correctly on Ubuntu. I've open sourced my code here: https://github.com/DengYiping/Shuoshuo_crawler/ When I use clang to compile:

scott@Harold:~/Apps/shuoshuo_crawler/shuoshuo_crawler/qqlogin$ make
clang -c -Wall -std=c++11 -stdlib=libc++ main.cpp -pthread
main.cpp:31:1: warning: unused label 'error_cleanup' [-Wunused-label]
error_cleanup:
^~~~~~~~~~~~~~
1 warning generated.
clang -c -Wall -std=c++11 -stdlib=libc++ qqlogin.cpp -pthread
clang -c -Wall -std=c++11 -stdlib=libc++ fetcher.cpp -pthread
clang -v  -Wl,--start-group -lmongoclient -lcurl -lpthread -ljsoncpp -lboost_system -lboost_thread -lboost_filesystem -lboost_program_options \-lssl -lcrypto  main.o qqlogin.o fetcher.o -o qq_crawler
Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4)
Target: x86_64-pc-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9.3
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5.2.1
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.3
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5.2.1
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
 "/usr/bin/ld" -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o qq_crawler /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crt1.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtbegin.o -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../.. -L/lib -L/usr/lib --start-group -lmongoclient -lcurl -lpthread -ljsoncpp -lboost_system -lboost_thread -lboost_filesystem -lboost_program_options -lssl -lcrypto main.o qqlogin.o fetcher.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtend.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld: //usr/local/lib/libmongoclient.a(logstream_builder.o): undefined reference to symbol '__tls_get_addr@@GLIBC_2.3'
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2: error adding symbols: DSO missing from command line
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [qq_crawler] Error 1

This is definitely a linker error, but I have no way to troubleshoot it.

My Makefile when using clang:

CC=clang
CFLAGS=-c -Wall -std=c++11 -stdlib=libc++
LDFLAGS= -v  -Wl,--start-group -lmongoclient -lcurl -lpthread -ljsoncpp -lboost_system -lboost_thread -lboost_filesystem -lboost_program_options \-lssl -lcrypto
EXECUTABLE=qq_crawler

all: qq_crawler

qq_crawler: main.o qqlogin.o fetcher.o
        $(CC) $(LDFLAGS) main.o qqlogin.o fetcher.o -o qq_crawler

main.o: main.cpp
        $(CC) $(CFLAGS) main.cpp -pthread

fetcher.o: fetcher.cpp
        $(CC) $(CFLAGS) fetcher.cpp -pthread

qqlogin.o: qqlogin.cpp
        $(CC) $(CFLAGS) qqlogin.cpp -pthread
clean:
        rm *.o qq_crawler

Then I try to use g++ to compile, and this is my Makefile:

CC=g++-4.9
CFLAGS=-c -Wall -std=c++11
LDFLAGS= -v  -Wl,--start-group -lmongoclient -lcurl -lpthread -ljsoncpp -lboost_system -lboost_thread -lboost_filesystem -lboost_program_options \-lssl -lcrypto
EXECUTABLE=qq_crawler

all: qq_crawler

qq_crawler: main.o qqlogin.o fetcher.o
        $(CC) $(LDFLAGS) main.o qqlogin.o fetcher.o -o qq_crawler

main.o: main.cpp
        $(CC) $(CFLAGS) main.cpp -pthread

fetcher.o: fetcher.cpp
        $(CC) $(CFLAGS) fetcher.cpp -pthread

qqlogin.o: qqlogin.cpp
        $(CC) $(CFLAGS) qqlogin.cpp -pthread
clean:
        rm *.o qq_crawler

And this is result. The result shows that condition_variables are not available.

g++-4.9 -c -Wall -std=c++11 main.cpp -pthread
In file included from fetcher.hpp:22:0,
                 from main.cpp:11:
threadtool.h:22:10: error: ‘condition_variable’ in namespace ‘std’ does not name a type
     std::condition_variable queue_cond;
          ^
threadtool.h: In member function ‘void threadtool::Threadsafe_queue<T>::push(T)’:
threadtool.h:38:7: error: ‘queue_cond’ was not declared in this scope
       queue_cond.notify_one();
       ^
threadtool.h: In member function ‘void threadtool::Threadsafe_queue<T>::wait_pop(T&)’:
threadtool.h:60:7: error: ‘queue_cond’ was not declared in this scope
       queue_cond.wait(locker,[this]{return !raw_queue.empty();});
       ^
threadtool.h: In member function ‘std::shared_ptr<_Tp1> threadtool::Threadsafe_queue<T>::wait_pop()’:
threadtool.h:68:7: error: ‘queue_cond’ was not declared in this scope
       queue_cond.wait(locker,[this]{return !raw_queue.empty();});
       ^
threadtool.h: At global scope:
threadtool.h:101:10: error: ‘condition_variable’ in namespace ‘std’ does not name a type
     std::condition_variable stack_cond;
          ^
threadtool.h: In member function ‘void threadtool::Threadsafe_stack<T>::push(T)’:
threadtool.h:116:7: error: ‘stack_cond’ was not declared in this scope
       stack_cond.notify_one();
       ^
threadtool.h: In member function ‘void threadtool::Threadsafe_stack<T>::wait_pop(T&)’:
threadtool.h:126:7: error: ‘stack_cond’ was not declared in this scope
       stack_cond.wait(locker,[this]{return !raw_stack.empty();});
       ^
threadtool.h: In member function ‘std::shared_ptr<_Tp1> threadtool::Threadsafe_stack<T>::wait_pop()’:
threadtool.h:134:7: error: ‘stack_cond’ was not declared in this scope
       stack_cond.wait(locker,[this]{return !raw_stack.empty();});
       ^
main.cpp: In function ‘int main(int, const char**)’:
main.cpp:31:1: warning: label ‘error_cleanup’ defined but not used [-Wunused-label]
 error_cleanup:
 ^
make: *** [main.o] Error 1

Then I try to use clang to compile and gcc to link the compiled .o file.

g++-4.9 -v  -Wl,--start-group -lmongoclient -lcurl -lpthread -ljsoncpp -lboost_system -lboost_thread -lboost_filesystem -lboost_program_options \-lssl -lcrypto main.o qqlogin.o fetcher.o -o qq_crawler
Using built-in specs.
COLLECT_GCC=g++-4.9
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.9.3-5ubuntu1~14.04' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.3 (Ubuntu 4.9.3-5ubuntu1~14.04) 
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-o' 'qq_crawler' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.9/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/4.9/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper -plugin-opt=-fresolution=/tmp/cc01qoam.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=/ --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro -o qq_crawler /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.9/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../.. --start-group -lmongoclient -lcurl -lpthread -ljsoncpp -lboost_system -lboost_thread -lboost_filesystem -lboost_program_options -lssl -lcrypto main.o qqlogin.o fetcher.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/4.9/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld: //usr/local/lib/libmongoclient.a(replica_set_monitor.o): undefined reference to symbol 'ceilf@@GLIBC_2.2.5'
/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/libm.so: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [qq_crawler] Error 1

I've open source my code here: https://github.com/DengYiping/Shuoshuo_crawler/

Thank you so much for your support.

Answer 1

There are various things wrong with your makefiles, both with CC = clang and CC = g++-4.9

The foremost error with CC = clang is that you are trying to build a C++ project
with a C compiler. clang is the LLVM C compiler. clang++ is the LLVM C++ compiler. That is the cause of the undefined symbol references when you compile with clang.

Though not the cause of any error here, convention dictates that makefiles denote the C compiler as CC and the C++ compiler as CXX, so your CC should become CXX.

Likewise the C compiler flags are denoted by CFLAGS and the C++ compiler flags as CXXFLAGS, so your CFLAGS should become CXXFLAGS.

The linker option that requests POSIX threads is -pthread, not -lpthread.

Libraries required for linkage should be denoted by the variable LIBS, and other linker options by LDFLAGS. You don't make this distinction and include all your -l<libname> options in LDFLAGS. Then you wish to link your executable with the recipe:

$(CC) $(LDFLAGS) main.o qqlogin.o fetcher.o -o qq_crawler

In the resulting commandline, this will make all of your libraries appear before any of your object files, so the libraries will be ignored and the linkage will fail with unresolved symbols. In the commandline sequence of things to be linked, things that need symbol definitions must appear before the things that provide the definitions. That is why you need:

$(CC) $(LDFLAGS) main.o qqlogin.o fetcher.o -o qq_crawler $(LIBS)

The code you have published at https://github.com/DengYiping/Shuoshuo_crawler/ is evidently not up to date, because the makefile there is different from either of those that you have posted here and in particular it refers to an object file downloader.o for which there is no source file: it ought to be qqlogin.o. However, if the code you have published requires at least the same libraries as the code you are talking about here then there are two libraries required for linkage that are not mentioned in your makefile: libboost_regex and libboost_system

There is a bug in your clean target:

clean:
    rm *.o qq_crawler

It should be:

clean:
    rm -f *.o qq_crawler

With out the -f, if make clean is invoked when the object files or qq_crawler do not exist for any reason then make clean will fail.

As you are linking libjsoncpp you need to have installed libjsoncpp-dev, or else built and installed the library yourself. If you have installed libjsoncpp-dev then your compilation on Ubuntu will require the include-search option:

-I/usr/include/jsoncpp

Alternatively, you can replace #include <json/json.h> with #include <jsoncpp/json/json.h> in your code.

There are no more major defects in your makefile.

The libmongo-client distributed with Ubuntu <= 15.10 is too old for your build. Therefore you need to build it yourself and either install it under /usr/local or else provide a compiler option -I/path/to/mongo/headers and a linker option -L/path/to/mongo/libs. From the linker error messages you have posted, e.g.

/usr/bin/ld: //usr/local/lib/libmongoclient.a(logstream_builder.o): undefined reference to symbol '__tls_get_addr@@GLIBC_2.3'

I conclude that you have already discovered this and have installed the current version or the one you need under /usr/local/, so you will not need an -I or -L option for libmongoclient.

There is no need for you to compile with Clang rather than GCC. GCC is not the cause of any problem.

Putting all this together, you can build the project published at https://github.com/DengYiping/Shuoshuo_crawler/ with this makefile:

CXX=g++
CXXFLAGS=-c -Wall -std=c++11 -I/usr/include/jsoncpp
LDFLAGS=-pthread
LIBS= -lcurl -lstdc++ -lpthread -ljsoncpp -lmongoclient -lboost_thread -lboost_filesystem -lboost_program_options -lboost_regex -lboost_system

all: qq_crawler

qq_crawler: main.o qqlogin.o fetcher.o
    $(CXX) $(LDFLAGS) main.o qqlogin.o fetcher.o -o qq_crawler $(LIBS)

main.o: main.cpp
    $(CXX) $(CXXFLAGS) main.cpp

fetcher.o: fetcher.cpp
    $(CXX) $(CXXFLAGS) fetcher.cpp

qqlogin.o: qqlogin.cpp
    $(CXX) $(CXXFLAGS) qqlogin.cpp
clean:
    rm -f *.o qq_crawler

It is not a praiseworthy makefile, but it will do.

Before you can do that, however, you need to fix the various compilation errors you encountered in threadtool.h when building with g++-4.9. These are caused by the fact that in threadtool.h you have not included the Standard header in which std::condition_variable is declared; so edit threadtool.h and add

#include <condition_variable>

after #include <stack>.

Now the project will build with the makefile above, assuming the required libraries are appropriately installed.

Finally, as the build proceeds you will observe numerous occurrences of the warning:

warning: ‘template<class> class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]

As it says, std::auto_ptr is deprecated in C++11. That is because std::auto_ptr was a a fundamentally flawed attempt at a smart pointer. Deprecation means that in future Standard libraries it will disappear. You should remove it from your code and replace it with either std::shared_pointer or std::unique_ptr, or a combination of the two. Read the documentation and decide which replacement is right for each of your std::auto_ptrs.

Later

libmongoclient is actually compiled using clang with libc++ and c++11 on. It should work correctly. I still face some problem.

It won't work correctly. You need to download and build mongo-client from source (with GCC, default), and install to /usr/local:

$ sudo apt-get install scons
$ git clone https://github.com/mongodb/mongo-cxx-driver.git
$ cd mongo-cxx-driver
$ scons CXXFLAGS=-Wno-error=unused-variable
$ sudo scons install --prefix=/usr/local