clang memory sanitizer (linux) does not report errors

Question

I'm attempting to produce clang memory sanitizer errors on a linux platform with a toy program. I haven't been able to produce any errors with what I believe is a correct platform / compilation / execution.

A simple description of the platform:

• debian 8 in a docker container
• standard clang package (3.5)
• custom (msan) libc++ / libc++abi compiled using these instructions

The problem is that a simple test program does not report an error. This is the test program:

#include <iostream>
#include <string>

int main()
{
    auto s = std::string("asdf");
    if (s[5] == 'z')
        std::cout << "asdf\n";
    else
        std::cout << "qwer\n";
    return 0;
}

Therein, uninitialized memory is read. This is how it was compiled:

user@29a0fe911f05:/tmp$ clang++ -v -Wl,-rpath=/tmp/libcxx_msan/lib/ -I /tmp/libcxx_msan/include/ -I /tmp/libcxx_msan/include/c++/v1 -std=c++11 -stdlib=libc++ -fsanitize=memory test.cc
Debian clang version 3.5.0-10 (tags/RELEASE_350/final) (based on LLVM 3.5.0)
Target: x86_64-pc-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9.2
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.2
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/lib/llvm-3.5/bin/clang" -cc1 -triple x86_64-pc-linux-gnu -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -main-file-name test.cc -mrelocation-model pic -pic-level 2 -pie-level 2 -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -target-linker-version 2.25 -v -dwarf-column-info -resource-dir /usr/lib/llvm-3.5/bin/../lib/clang/3.5.0 -I /tmp/libcxx_msan/include/ -I /tmp/libcxx_msan/include/c++/v1 -internal-isystem /usr/include/c++/v1 -internal-isystem /usr/include/clang/3.5.0/include/ -internal-isystem /usr/local/include -internal-isystem /usr/lib/llvm-3.5/bin/../lib/clang/3.5.0/include -internal-externc-isystem /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++11 -fdeprecated-macro -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 239 -fsanitize=memory -fno-assume-sane-operator-new -mstackrealign -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/test-548974.o -x c++ test.cc
clang -cc1 version 3.5.0 based upon LLVM 3.5.0 default target x86_64-pc-linux-gnu
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/include/clang/3.5.0/include"
#include "..." search starts here:
#include <...> search starts here:
 /tmp/libcxx_msan/include
 /tmp/libcxx_msan/include/c++/v1
 /usr/include/c++/v1  
 /usr/include/clang/3.5.0/include
 /usr/local/include
 /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/usr/bin/ld" -whole-archive /usr/lib/llvm-3.5/bin/../lib/clang/3.5.0/lib/linux/libclang_rt.msan-x86_64.a -no-whole-archive -pie --hash-style=both --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/Scrt1.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtbeginS.o -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../.. -L/usr/lib/llvm-3.5/bin/../lib -L/lib -L/usr/lib -rpath=/tmp/libcxx_msan/lib/ /tmp/test-548974.o -lpthread -lrt -lm -ldl -export-dynamic -lc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtendS.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o

The program is using libc++ / libc++abi libraries compiled using the instructions linked in the platform description.

user@29a0fe911f05:/tmp$ ldd a.out
        linux-vdso.so.1 (0x00007ffeadbb8000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5477040000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5476e38000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5476b30000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5476928000)
        libc++.so.1 => /tmp/libcxx_msan/lib/libc++.so.1 (0x00007f5476580000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5476368000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5475fb8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5479950000)
        libc++abi.so.1 => /tmp/libcxx_msan/lib/../lib/libc++abi.so.1 (0x00007f5475cc0000)

This is the program's output:

user@29a0fe911f05:/tmp$ ./a.out 
qwer

I can produce clang memory sanitizer false-positive errors with an incorrect platform configuration, which I understand is expected behavior. When the program is linked against uninstrumented libc++ / libc++abi, expected false-positives are produced.

user@29a0fe911f05:/tmp$ clang++ -v -std=c++11 -stdlib=libc++ -fsanitize=memory test.cc
Debian clang version 3.5.0-10 (tags/RELEASE_350/final) (based on LLVM 3.5.0)
Target: x86_64-pc-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9.2
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.2
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/lib/llvm-3.5/bin/clang" -cc1 -triple x86_64-pc-linux-gnu -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -main-file-name test.cc -mrelocation-model pic -pic-level 2 -pie-level 2 -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -target-linker-version 2.25 -v -dwarf-column-info -resource-dir /usr/lib/llvm-3.5/bin/../lib/clang/3.5.0 -internal-isystem /usr/include/c++/v1 -internal-isystem /usr/include/clang/3.5.0/include/ -internal-isystem /usr/local/include -internal-isystem /usr/lib/llvm-3.5/bin/../lib/clang/3.5.0/include -internal-externc-isystem /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++11 -fdeprecated-macro -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 239 -fsanitize=memory -fno-assume-sane-operator-new -mstackrealign -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/test-9f8b7c.o -x c++ test.cc
clang -cc1 version 3.5.0 based upon LLVM 3.5.0 default target x86_64-pc-linux-gnu
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/include/clang/3.5.0/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/include/c++/v1
 /usr/include/clang/3.5.0/include
 /usr/local/include
 /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/usr/bin/ld" -whole-archive /usr/lib/llvm-3.5/bin/../lib/clang/3.5.0/lib/linux/libclang_rt.msan-x86_64.a -no-whole-archive -pie --hash-style=both --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/Scrt1.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtbeginS.o -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../.. -L/usr/lib/llvm-3.5/bin/../lib -L/lib -L/usr/lib /tmp/test-9f8b7c.o -lpthread -lrt -lm -ldl -export-dynamic -lc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtendS.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o
user@29a0fe911f05:/tmp$ ldd a.out
        linux-vdso.so.1 (0x00007ffc09be0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3e6c730000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f3e6c528000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3e6c220000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3e6c018000)
        libc++.so.1 => /usr/lib/x86_64-linux-gnu/libc++.so.1 (0x00007f3e6bd10000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3e6baf8000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3e6b748000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3e6f048000)
user@29a0fe911f05:/tmp$ ./a.out 
==3014== WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7fcb58194a89 in main (/tmp/a.out+0x94a89)
    #1 0x7fcb56cf1b44 in __libc_start_main /build/glibc-I9DIZl/glibc-2.19/csu/libc-start.c:287
    #2 0x7fcb5819427c in _start (/tmp/a.out+0x9427c)

SUMMARY: MemorySanitizer: use-of-uninitialized-value ??:0 main
Exiting

Answer 1

Short answer: string is a special case.

Longer answer: in general, std::string is the same size as three pointers. One points to the start of the string, one points to the end, and the third points to the end of the allocated space. (begin, end, and capacity). in libc++, the implementation of std::string uses the "short string optimization", where if the contents of the string are short, then the data is stored in the 24 bytes (for x86_64) that the pointers would take. This eliminates a heap allocation for many strings.

My understanding is that memory sanitizer doesn't know this about std::string, and so it appears that those bytes have been written - even though they're past the end of the string's data.

What happens if you change your test string to "abcdefghijklmnopqrstuvwxyz0123456789" (and then attempt to access element 37)?

P.S. The sanitizer team has recently sent me a patch to add ASAN support to libc++'s string implementation. I'll ask about MSAN, too.