ThreadSanitizer (tsan) - Meaningful information from shared library

Question

My app used the wxWidgets library, built from source via gcc 5.1.0, using -g and -O0

I compiled my app using clang++36 -g -fsanitize=thread -stdlib=libc++, and linked it using clang++36 -g -fsanitize=thread -stdlib=libc++ -lc++abi. It dynamically links to wxWidgets.

One of the warnings I received was:

==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=52741)
  Cycle in lock order graph: M115 (0x7d080000ea60) => M976 (0x7d0800000100) => M115

  Mutex M976 acquired here while holding mutex M115 in main thread:
    #0 pthread_mutex_lock /home/xxx/sourceInstallations/llvm-3.6.0/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:3008 (wxDebugSleep+0x00000043b0ef)
    #1 <null> <null> (libwx_baseu-3.0.so.0+0x0000002376fa)
    #2 _start <null> (wxDebugSleep+0x00000041be4e)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M115 acquired here while holding mutex M976 in main thread:
    #0 pthread_mutex_lock /home/xxx/sourceInstallations/llvm-3.6.0/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:3008 (wxDebugSleep+0x00000043b0ef)
    #1 <null> <null> (libwx_baseu-3.0.so.0+0x0000002376fa)
    #2 wxCriticalSectionLocker::wxCriticalSectionLocker(wxCriticalSection&) /usr/local/include/wx-3.0/wx/thread.h:307:9 (wxDebugSleep+0x000000473216)
    #3 <null> <null> (libwx_baseu-3.0.so.0+0x00000018b297)
    #4 _start <null> (wxDebugSleep+0x00000041be4e)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 ??
==================

I wasn't happy because: (1) I'd like to have a shot at finding threading errors in the wxWidgets library; and (2) I'd like to be able to make a suppression file operating at or near a one warning to suppression file ratio.

So, I recompiled/linked the wxWidgets library from source via clang 3.6.0, adding -fsanitize=thread -stdlib=libc++ -lc++abi. Crossed my fingers, and it finished just fine.

Ran sudo make uninstall in my wxWidgets gcc build directory, and sudo install in my wxWidgets clang build directory.

The warning above now shows:

==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=68453)
  Cycle in lock order graph: M115 (0x7d080000ea60) => M976 (0x7d0800000100) => M115

  Mutex M976 acquired here while holding mutex M115 in main thread:
    #0 pthread_mutex_lock /home/xxx/sourceInstallations/llvm-3.6.0/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:3008 (wxDebugSleep+0x00000043b0ef)
    #1 <null> <null> (libwx_baseu-3.0.so.0+0x0000003c24f9)
    #2 <null> <null> (libwx_baseu-3.0.so.0+0x0000003d0387)
    #3 <null> <null> (libwx_baseu-3.0.so.0+0x0000003d0000)
    #4 <null> <null> (libwx_baseu-3.0.so.0+0x0000001bd91c)
    #5 <null> <null> (libwx_baseu-3.0.so.0+0x000000279cd6)
    #6 <null> <null> (libwx_baseu-3.0.so.0+0x00000027a6da)
    #7 <null> <null> (libwx_baseu-3.0.so.0+0x00000024445d)
    #8 <null> <null> (libwx_baseu-3.0.so.0+0x000000244243)
    #9 <null> <null> (libwx_baseu-3.0.so.0+0x000000245a67)
    #10 <null> <null> (libwx_baseu-3.0.so.0+0x000000246856)
    #11 <null> <null> (libwx_baseu-3.0.so.0+0x000000245430)
    #12 <null> <null> (libwx_baseu-3.0.so.0+0x000000245934)
    #13 main /home/xxx/code/testing/wxDebugSleep/wxDebugSleep.cpp:11:1 (wxDebugSleep+0x000000472e9c)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M115 acquired here while holding mutex M976 in main thread:
    #0 pthread_mutex_lock /home/xxx/sourceInstallations/llvm-3.6.0/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:3008 (wxDebugSleep+0x00000043b0ef)
    #1 <null> <null> (libwx_baseu-3.0.so.0+0x0000003c24f9)
    #2 <null> <null> (libwx_baseu-3.0.so.0+0x0000003d0387)
    #3 wxCriticalSection::Enter(void) /usr/local/include/wx-3.0/wx/thread.h:291:52 (wxDebugSleep+0x00000047c570)
    #4 wxCriticalSectionLocker::wxCriticalSectionLocker(wxCriticalSection&) /usr/local/include/wx-3.0/wx/thread.h:307:9 (wxDebugSleep+0x000000473216)
    #5 <null> <null> (libwx_baseu-3.0.so.0+0x000000245cf0)
    #6 <null> <null> (libwx_baseu-3.0.so.0+0x000000246949)
    #7 <null> <null> (libwx_baseu-3.0.so.0+0x00000024574b)
    #8 <null> <null> (libwx_baseu-3.0.so.0+0x000000245934)
    #9 main /home/xxx/code/testing/wxDebugSleep/wxDebugSleep.cpp:11:1 (wxDebugSleep+0x000000472e9c)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 ??
==================

I defined TSAN_OPTIONS=second_deadlock_stack=1 in the environment of the running program, and it didn't change the output.

Well, it's some progress. I'm sure I'm going to use the wrong term, but it's like it's missing a symbols file for the library.

I've checked that it's dynamically linking against the new library using clang & -fsanitize=thread (ldd, and timestamps.)

I've checked that the library is compiling with -g and -O0 (even though it could be higher.)

In case it matters, FreeBSD 10.1 64-bit. Clang is compiled from source.

Question 1 - How can I get the "stack trace" from the shared library to show filename and line numbers?

Question 2 - If I can't, how can I make a good suppression file? The problem is that wxWidgets calls a lot of my code, so I don't think I can block any stack including the library. And, of course, even if I could make a suppression file using the offsets, if I recompile the library, all those could change.

Answer 1

The issue is that FreeBSD 10.1 and before has a bug, preventing llvm-symbolizer from working correctly. llvm-symbolizer is how TSAN gets the symbol information. More specifically, FreeBSD gives a dlpi_name without the path in dl_iterate_phdr. This was patched at https://reviews.freebsd.org/D932. The patch is available in FreeBSD 10-STABLE (at the time of this answer is like 10.2 beta) and should be in FreeBSD 10.2 and beyond.

BTW, "-Wl,--version-script..." and "-Wl,-soname" work fine.