Undefined reference to function although linked against library

Question

I am trying to build the open source project he-transformer in a Docker container. This is a tool build on top of ngraph, which is connected via ngraph-bridge with tensorflow, to enable writing deep learning programs using a graph-based abstraction. he-transformer is an extension of ngraph that allows inference over encrypted data using homomorphic encryption.

I use the latest he-transformer commit on the master branch (6d5a5b39). The build process throws many undefined reference errors:

#29 282.5 /usr/bin/clang++-9   -O2 -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -fstack-protector-all -Wall -march=native -Wno-deprecated-declarations -fopenmp=libomp  -O2 -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -fstack-protector-all -Wall -march=native -Wno-deprecated-declarations -g   -pie -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_subtract_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_slice_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_softmax_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_server_client_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_reverse_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_result_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_reshape_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_relu_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_read_write_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_power_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_pad_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_negate_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_max_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_multiply_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_minimum_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_maxpool_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_layers_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_rescale_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_dot_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_exp_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_divide_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_convolution_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_constant_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_concat_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_broadcast_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_basics_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_batch_norm_inference_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_avg_pool_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_add_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/main.cpp.o CMakeFiles/unit-test.dir/test_seal.cpp.o CMakeFiles/unit-test.dir/test_encryption_parameters.cpp.o CMakeFiles/unit-test.dir/test_he_op_annotations.cpp.o CMakeFiles/unit-test.dir/test_he_plaintext.cpp.o CMakeFiles/unit-test.dir/test_he_tensor.cpp.o CMakeFiles/unit-test.dir/test_he_type.cpp.o CMakeFiles/unit-test.dir/test_he_util.cpp.o CMakeFiles/unit-test.dir/test_he_fusion.cpp.o CMakeFiles/unit-test.dir/test_he_supported_ops.cpp.o CMakeFiles/unit-test.dir/test_propagate_he_annotations.cpp.o CMakeFiles/unit-test.dir/test_he_seal_executable.cpp.o CMakeFiles/unit-test.dir/test_bounded_relu.cpp.o CMakeFiles/unit-test.dir/test_perf_micro.cpp.o CMakeFiles/unit-test.dir/test_protobuf.cpp.o CMakeFiles/unit-test.dir/test_seal_plaintext_wrapper.cpp.o CMakeFiles/unit-test.dir/test_seal_util.cpp.o CMakeFiles/unit-test.dir/test_tcp_message.cpp.o CMakeFiles/unit-test.dir/test_tcp_client.cpp.o CMakeFiles/unit-test.dir/test_ngraph_he_log.cpp.o  -o unit-test -Wl,-rpath,/home/he-transformer/build/src:/home/he-transformer/build/ext_ngraph/src/ext_ngraph-build/src/ngraph:/home/he-transformer/build/ext_zlib/src/ext_zlib-build ../ext_ngraph/src/ext_ngraph-build/test/util/libngraph_test_util.a -lpthread ../src/libhe_seal_backend.so ../protobuf/lib/libprotobuf.a ../gtest/build/googlemock/gtest/libgtest.a ../ext_ngraph/src/ext_ngraph-build/src/ngraph/libngraph.so ../external/lib/libseal-3.4.a ../ext_zlib/src/ext_zlib-build/libz.so 
#29 282.6 INFO: From Compiling tensorflow/lite/kernels/mfcc.cc:
#29 282.6 In file included from ./tensorflow/lite/kernels/internal/common.h:42:0,
#29 282.6                  from ./tensorflow/lite/kernels/cpu_backend_gemm_custom_gemv.h:41,
#29 282.6                  from ./tensorflow/lite/kernels/cpu_backend_gemm.h:22,
#29 282.6                  from ./tensorflow/lite/kernels/internal/optimized/optimized_ops.h:42,
#29 282.6                  from tensorflow/lite/kernels/mfcc.cc:21:
#29 282.6 external/arm_neon_2_x86_sse/NEON_2_SSE.h: In function 'void vst1q_lane_f32(float32_t*, float32x4_t, int)':
#29 282.6 external/arm_neon_2_x86_sse/NEON_2_SSE.h:9725:31: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
#29 282.6      *(ptr) =  *((float*)&ilane);
#29 282.6                                ^
#29 282.6 external/arm_neon_2_x86_sse/NEON_2_SSE.h: In function 'float32_t vgetq_lane_f32(float32x4_t, int)':
#29 282.6 external/arm_neon_2_x86_sse/NEON_2_SSE.h:11964:22: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
#29 282.6      return *(float*)&ilane;
#29 282.6                       ^
#29 283.3 /home/he-transformer/build/test/test_sum_HE_SEAL.cpp:35: undefined reference to `ngraph::runtime::Backend::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)'
#29 283.3 /home/he-transformer/build/test/test_sum_HE_SEAL.cpp:51: undefined reference to `ngraph::Node::get_name[abi:cxx11]() const'
#29 283.3 CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o: In function `bool ngraph::runtime::he::test::all_close<float>(std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&, float)':
#29 283.3 /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:264: undefined reference to `ngraph::default_logger_handler_func(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 283.3 CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o: In function `bool ngraph::runtime::he::test::all_close<float>(std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&, float)':
#29 283.3 /home/he-transformer/test/./test_util.hpp:57: undefined reference to `ngraph::LogHelper::LogHelper(ngraph::LOG_TYPE, char const*, int, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)'
#29 283.3 CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o: In function `void __gnu_cxx::new_allocator<ngraph::Function>::construct<ngraph::Function, std::shared_ptr<ngraph::op::v0::Sum>&, std::vector<std::shared_ptr<ngraph::op::v0::Parameter>, std::allocator<std::shared_ptr<ngraph::op::v0::Parameter> > > >(ngraph::Function*, std::shared_ptr<ngraph::op::v0::Sum>&, std::vector<std::shared_ptr<ngraph::op::v0::Parameter>, std::allocator<std::shared_ptr<ngraph::op::v0::Parameter> > >&&)':
#29 283.3 /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/ext/new_allocator.h:136: undefined reference to `ngraph::Function::Function(std::shared_ptr<ngraph::Node> const&, std::vector<std::shared_ptr<ngraph::op::v0::Parameter>, std::allocator<std::shared_ptr<ngraph::op::v0::Parameter> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 283.3 CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o: In function `__cxx_global_var_init':
#29 283.3 /home/he-transformer/build/test/test_sum_HE_SEAL.cpp:65: undefined reference to `ngraph::combine_test_backend_and_case(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 283.3 /home/he-transformer/build/test/test_sum_HE_SEAL.cpp:65: undefined reference to `ngraph::prepend_disabled(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
...
#29 285.2 ../src/libhe_seal_backend.so: undefined reference to `ngraph::Node::get_provenance_tags[abi:cxx11]() const'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::file_util::read_file_to_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::to_lower(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::runtime::Backend::get_backend_op(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ...)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::runtime::BackendManager::register_backend(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<std::shared_ptr<ngraph::runtime::Backend> (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::file_util::exists(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::descriptor::Tensor::get_name[abi:cxx11]() const'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::descriptor::Tensor::Tensor(ngraph::element::Type const&, ngraph::PartialShape const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::split(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char, bool)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::Node::description[abi:cxx11]() const'
...
#29 285.3 clang: error: linker command failed with exit code 1 (use -v to see invocation)
#29 285.3 make[2]: *** [test/unit-test] Error 1
#29 285.3 test/CMakeFiles/unit-test.dir/build.make:825: recipe for target 'test/unit-test' failed
#29 285.3 make[2]: Leaving directory '/home/he-transformer/build'
#29 285.3 make[1]: *** [test/CMakeFiles/unit-test.dir/all] Error 2

Please see raw output of this gist for the full log and this repository for the Dockerfile I use to build he-transformer. (Note: I know that he-transformer by itself also provides a Dockerfile + Makefile but they did not work for me. However, mine is based on their Ubuntu 18.04 Dockerfile.)

I am not very familiar with cmake and the project's build setup is very complex with many dependencies but it sounds like libhe_seal_backend.so does not properly link against ngraph that is also build as external project, see ngraph.cmake. However, it is linked as you can see in this CMakeLists.txt in that the libhe_seal_backend target is built.

As the target unit-test causes the issues I thought that it might be required to link it against ngraph. For that, I added the respective link targets and replaced the file after cloning the he-transformer repository by the modified CMakeLists.txt. This unfortunately throws the same error.

NOTE: In case you are trying to reproduce this - it took me around 45 minutes to build it on a 48-cores AWS machine because it also builds tensorflow as external project.

Answer 1

Kudos for this hint to the repository maintainer of he-transformer, Fabian Boemer. He noticed that the logs indicate an issue with the CXX ABI. The issue is caused by compiling ngraph with an old version of g++ (< 5.1). The log confirms this, see compiler identification is GNU 4.8.5.

Removing the installation of gcc 4.8 from the Dockerfile so that the system's default is gcc-7 solved the issue and could successfully compile he-transformer.