CODEWITHSUNDEEP
c++clang++ulimitaddress-sanitizerlibfuzzer
Trevor Hickey
Trevor Hickey

Reputation: 37834

why does ulimit -v not work under clang's address sanitizer?

I'm using libFuzzer to fuzz an API.
The API is deserializing an array of bits (given by libFuzzer)
and converting them into c++ class instantiations.

Due to the serialization format, libFuzer is able construct a serialized object that tell the deserializer to reserve large amounts of data (which cannot be met).
This is done through calls to std::vector::resize(). The vector throws a std::bad_alloc, and although the problem is caught and safely mitigated, it causes extreme lag in the fuzzer (as mentioned in the following documentation on OOM issues).

In an attempt to lower the amount of memory used when the fuzzer is running, I was hoping to set ulimit -v and adjust the available virtual memory of the process. However doing so causes 

==27609==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)
==27609==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v

Why can't the address sanitizer work under ulmit -v?
I wish it could, then I might be able to fuzz more effectively.

Other information:
My build flags were: 

copts = [
    "-fsanitize=address,fuzzer",
    "-fsanitize-trap=undefined,integer",
    "-fsanitize-coverage=trace-pc,trace-cmp,trace-pc-guard",
    "-g",
    "-O0",
    "-fno-omit-frame-pointer",
    "-fno-sanitize=vptr",
],
linkopts = [
    "-fsanitize=address,fuzzer",
    "-fsanitize-trap=undefined,integer",
    "-fno-sanitize=vptr",
    "-fsanitize-link-c++-runtime",
],

I tried turning flags off so I could set ulimit and run the fuzzer: 

copts = [
    "-fsanitize=fuzzer",
    "-g",
    "-O0",
    "-fno-omit-frame-pointer",
],
linkopts = [
    "-fsanitize=fuzzer",
],

but this causes an immediate segfault.

Upvotes: 6

Views: 2945

Answers (1)

yugr
yugr

Reputation: 21916

Asan reserves 1/8-th of process address space for shadow memory at startup to hold status of user data (allocated, freed, etc.). This is by design and there is nothing one can do about it.

Note that you generally don't care about virtual memory but rather physical one (which is also causing new to fail in your case).

Upvotes: 2

Related Questions