CODEWITHSUNDEEP
c++macosllvmaddress-sanitizerlibc++
Alex O
Alex O

Reputation: 1876

Sanitizer crash inside libc++

A little background:

Our team's developers use mostly Macs while the target servers run Ubuntu. After being bitten by subtle compiler/library differences between the platforms, we decided to standardize on LLVM 17 across the board (clang, libc++, lldb, the works).

Prior to that, I was using Apple Clang from Xcode command-line tools, and still have it installed. After the switch, I am using LLVM@17 from Homebrew

The product consists of C++ code linked with several 3rd party libraries (grpc, redis, pqxx, etc) which were also built locally from source using the same tools. Our code is compiled as -std=c++23 but some of the libraries were compiled under a previous standard due to incompatibilities.

The problem:

When building an executable with sanitizers and running it locally, it crashes with the following output:

/opt/homebrew/opt/llvm@17/bin/../include/c++/v1/__charconv/from_chars_integral.h:46:16: runtime error: call to function std::__1::from_chars_result std::__1::__from_chars_atoi[abi:ue170006]<unsigned long long, 0>(char const*, char const*, unsigned long long&) through pointer to incorrect function type 'std::from_chars_result (*)(const char *, const char *, unsigned long long &)'
(server:arm64+0x10083d154): note: std::__1::from_chars_result std::__1::__from_chars_atoi[abi:ue170006]<unsigned long long, 0>(char const*, char const*, unsigned long long&) defined here
    #0 0x1011a4df4 in std::__1::from_chars_result std::__1::__sign_combinator[abi:ue170006]<char const*, long long, std::__1::from_chars_result (*)(char const*, char const*, unsigned long long&)>(char const*, char const*, long long&, std::__1::from_chars_result (*)(char const*, char const*, unsigned long long&))+0x278 (server:arm64+0x1011a4df4)
    #1 0x1011a4a7c in std::__1::from_chars_result std::__1::__from_chars_atoi[abi:ue170006]<long long, 0>(char const*, char const*, long long&)+0x188 (server:arm64+0x1011a4a7c)
    #2 0x102730a90 in pqxx::internal::integral_traits<long long>::from_string(std::__1::basic_string_view<char, std::__1::char_traits<char>>)+0x54 (server:arm64+0x102730a90)
    #3 0x100748320 in long long pqxx::from_string<long long>(std::__1::basic_string_view<char, std::__1::char_traits<char>>)+0x1b4 (server:arm64+0x100748320)
[... more stack frames here ...]
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /opt/homebrew/opt/llvm@17/bin/../include/c++/v1/__charconv/from_chars_integral.h:46:16 in

There are no such errors when running an executable without sanitizers.

Likewise, there are no such errors when the executable runs on Ubuntu (the target servers, the build environment, or devs using WSL2 under Windows)

Thoughts:

I suspect that this might have something to do with the executable loading two versions of the libc++ and libc++abi libraries (the Apple/Xcode ones and the Homebrew/LLVM ones).

When running under the debugger, I can see that both are loaded:

Stop reason: Function type mismatch
Stop reason: signal SIGABRT
image list
[  0] 4C4C44F0-5555-3144-A1BA-7CD98681E0B4 0x0000000100000000 /Users/oren/work/server-2/build/server/server
[  1] 4CE86D18-F3FA-3D2A-A1B8-E7CD8A52FB0D 0x000000018c0de000 /usr/lib/dyld 
[  2] 86BB7A29-1B92-38E5-A47D-7AD8FA8E9C19 0x00000001127bc000 /opt/homebrew/Cellar/llvm@17/17.0.6/lib/c++/libc++.1.0.dylib 
[  3] 59CB7CE8-1EB3-3DBE-98C8-DB08E836312D 0x0000000112c64000 /opt/homebrew/Cellar/llvm@17/17.0.6/lib/clang/17/lib/darwin/libclang_rt.asan_osx_dynamic.dylib 
... more lines ...
[ 42] 17A73968-9D77-3CC1-8382-84292D845617 0x0000000199fed000 /usr/lib/system/libunwind.dylib 
[ 47] 6477CE88-B069-395C-B258-7C6162AD5257 0x000000018c407000 /usr/lib/libc++abi.dylib 
[ 48] 6CCA803C-BBBE-3265-889E-AF4F60CAFD5D 0x000000026bee9000 /usr/lib/liboah.dylib 
[ 49] FE5F2FBB-D3B5-30BC-BAD8-3571AC2A72A0 0x000000018c379000 /usr/lib/libc++.1.dylib 
... many more lines ...
[301] 7AA0CBE1-6414-34F0-AE51-1071BE00C574 0x00000001125e0000 /opt/homebrew/Cellar/llvm@17/17.0.6/lib/c++/libc++abi.1.0.dylib

Also, when the debugger loads the program, it complains about dynamic cast errors:

2025-01-23 14:13:13.845486-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. St9type_info, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv117__class_type_infoE.
2025-01-23 14:13:13.846495-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. N10__cxxabiv117__class_type_infoE, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv120__si_class_type_infoE.
2025-01-23 14:13:13.846507-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. N10__cxxabiv117__class_type_infoE, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv120__si_class_type_infoE.
2025-01-23 14:13:13.846718-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. N10__cxxabiv117__class_type_infoE, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv120__si_class_type_infoE.
2025-01-23 14:13:13.846814-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. N10__cxxabiv117__class_type_infoE, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv120__si_class_type_infoE.
2025-01-23 14:13:13.847835-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. St9type_info, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv117__class_type_infoE.
2025-01-23 14:13:13.851894-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. N10__cxxabiv117__class_type_infoE, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv120__si_class_type_infoE.
2025-01-23 14:13:13.852247-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. St9type_info, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv117__class_type_infoE.
2025-01-23 14:13:13.888734-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. N10__cxxabiv117__class_type_infoE, N10__cxxabiv121__vmi_class_type_infoE, N10__cxxabiv121__vmi_class_type_infoE.
2025-01-23 14:13:13.915920-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. St9type_info, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv117__class_type_infoE.
[... program log entries here ...]
2025-01-23 14:13:14.010867-0500 server[13655:110398063] dynamic_cast error 2: One or more of the following type_info's has hidden visibility or is defined in more than one translation unit. They should all have public visibility. N10__cxxabiv117__class_type_infoE, N10__cxxabiv120__si_class_type_infoE, N10__cxxabiv120__si_class_type_infoE.
/opt/homebrew/opt/llvm@17/bin/../include/c++/v1/__charconv/from_chars_integral.h:46:16: runtime error: call to function std::__1::from_chars_result std::__1::__from_chars_atoi[abi:ue170006]<unsigned long long, 0>(char const*, char const*, unsigned long long&) through pointer to incorrect function type 'std::from_chars_result (*)(const char *, const char *, unsigned long long &)'
(server:arm64+0x10083d154): note: std::__1::from_chars_result std::__1::__from_chars_atoi[abi:ue170006]<unsigned long long, 0>(char const*, char const*, unsigned long long&) defined here
    #0 0x1011a4df4 in std::__1::from_chars_result std::__1::__sign_combinator[abi:ue170006]<char const*, long long, std::__1::from_chars_result (*)(char const*, char const*, unsigned long long&)>(char const*, char const*, long long&, std::__1::from_chars_result (*)(char const*, char const*, unsigned long long&))+0x278 (server:arm64+0x1011a4df4)
    #1 0x1011a4a7c in std::__1::from_chars_result std::__1::__from_chars_atoi[abi:ue170006]<long long, 0>(char const*, char const*, long long&)+0x188 (server:arm64+0x1011a4a7c)
    #2 0x102730a90 in pqxx::internal::integral_traits<long long>::from_string(std::__1::basic_string_view<char, std::__1::char_traits<char>>)+0x54 (server:arm64+0x102730a90)
    #3 0x100748320 in long long pqxx::from_string<long long>(std::__1::basic_string_view<char, std::__1::char_traits<char>>)+0x1b4 (server:arm64+0x100748320)
[... more stack frames here ...]
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /opt/homebrew/opt/llvm@17/bin/../include/c++/v1/__charconv/from_chars_integral.h:46:16 in

Those dynamic cast errors also do not appear when I build the executable without sanitizers. While the mere inclusion of sanitizers may be responsible for the crash, it would not explain why it never happens on Ubuntu.

For the same reason, I don't think that having compiled some of the libraries under a different version of C++ standard is to blame.

Curiously, while I still saw the dynamic cast errors when running MacOS 14, I did not encounter the actual crash until I upgraded to MacOS 15.2 (and likewise upgraded Xcode and the Xcode command line tools).

And to rule out the debugger as the culprit, the crash also happens when running the program from the shell.

I have currently disabled sanitizers for local builds, but I would rather fix (or work around) the issue instead.

Edit to address comments:

Building uses this cmake snippet:

if(ASAN_ENABLED)
    set(SANITIZE_FLAGS "-fsanitize=address")
    set(SANITIZE_FLAGS "${SANITIZE_FLAGS} -fsanitize=undefined") # Undefined behaviour
    set(SANITIZE_FLAGS "${SANITIZE_FLAGS} -fsanitize=local-bounds,unsigned-integer-overflow,implicit-conversion,float-divide-by-zero")
    set(SANITIZE_FLAGS "${SANITIZE_FLAGS} -fsanitize-ignorelist=${CMAKE_SOURCE_DIR}/CMake/ignorelist.txt") # suppressions
    set(SANITIZE_FLAGS "${SANITIZE_FLAGS} -fno-sanitize-recover=all") # crash on violations
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SANITIZE_FLAGS}")
    set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fno-omit-frame-pointer -fno-optimize-sibling-calls")
endif()

Upvotes: 0

Views: 82

Answers (0)

Related Questions