CODEWITHSUNDEEP
linuxc++11memory-leaks64-bitgcc4.7
Mike
Mike

Reputation: 647

Strange memory behavior with std map and shared_ptr

The code below provokes a strange memory behavior on my Debian machine. Even after maps are cleared, htop shows that the program still uses a lot of memory, which makes me think there is a memory leak. The strange fact is that it only appears under some circumstances.

#include <map>
#include <iostream>
#include <memory>


int main(int argc, char** argv)
{
    if (argc != 2)
    {
        std::cout << "Usage: " << argv[0] << " <1|0> " << std::endl;
        std::cout << "1 to insert in the second map and see the problem "
                "and 0 to not insert" << std::endl;
        return 0;
    }

    bool insertion = atoi(argv[1]);
    std::map<uint64_t, std::shared_ptr<std::string> > mapStd;
    std::map<uint64_t, size_t> counterToSize;
    size_t dataSize = 1024*1024;
    uint64_t counter = 0;

    while(counter < 10000)
    {
        std::shared_ptr<std::string> stringPtr =
                std::make_shared<std::string>(dataSize, 'a');
        mapStd[counter] = stringPtr;

        if (insertion)
        {
            counterToSize[counter] = dataSize;
        }
        if (counter > 500)
        {
            mapStd.erase(mapStd.begin());
        }
        std::cout << "\rInserted chunk " << counter << std::flush;

        counter++;
    }

    std::cout << std::endl << "Press ENTER to delete the maps" << std::endl;
    char a;
    std::cin.get(a); // wait for ENTER to be pressed

    mapStd.clear();   // clear both maps
    counterToSize.clear();

    std::cout << "Press ENTER to exit the program" << std::endl;

    std::cin.get(a); // wait for ENTER to be pressed
    return 0;
}

Explanation:

The code creates two maps on the stack (but issue is the same if they are created on the heap). It then inserts std::shared_ptr of strings to the first map. Each string has a size of 1MB. Once 500 strings are inserted, one deletes the first one for each new insert, so that the total memory used by the map is always equal to 500MB. When a total of 10000 strings have been inserted, the program waits for the user to press ENTER. If you launch the program and pass 1 as first argument, then, for each insertion in the first map, another insertion is also made to the second map. If the first argument is 0, then the second map is not used. Once ENTER is pressed for the first time, both maps are cleared. The program still runs and waits again for ENTER to be pressed, and then exits.

Here are the facts:

Does somebody have an explanation for all this ???

Upvotes: 3

Views: 972

Answers (1)

Charlie
Charlie

Reputation: 1592

I recommend looking here followed by here. Basically, the libc malloc starts out by using mmap for "large" allocations (> 128k), and brk/freelists for small allocations. Once one of the large allocations is free'd, it tries to adjust the sizes where it might use malloc, but only if the size is less than the max (defined at that first link). In the 32 bit case, your strings are way above the max, so it continues to use mmap/munmap for the large allocations, and only puts the smaller map node allocations into memory retrieved from the system with sbrk. That's why you don't see the 'problem' on 32 bit systems.

The other bit is one of fragmentation and when free tries to coalesce memory and return it to the system. By default, free will just stick small blocks onto a free list so that they're ready to go for the next request. If a large enough block is free'd at the top of the heap, it will try to return memory to the system see comment here. The threshold is 64K.

Your allocation pattern in case where you pass 1 likely leaves some element of the counterToSize map near the top of the heap preventing it from being free'd by the last release of one of the strings. The releases of the various objects inside of the counterToSize map are not large enough to trigger the thresholds.

If you switch the order of your .clear() calls, you'll find that the memory is released like you'd expect. Also, if you were to allocate a large chunk of memory and immediately free it immediately after the clears, it'd trigger the release. (Large in this case just needs to be more than 128 bytes - the max sized used to trigger fast bins. (i.e. free's of that size and allocs less than that size just go onto the list.

I hope that was clear. Basically, it's not really a problem. You've got some pages mapped. You're not using anything on them, but the last free that might have released them was too small to trigger that code path. The next time you try to allocate something, it will pull from the memory you already have (you could do the whole loop again without growing memory).

Oh, and you could call malloc_trim() by hand and force it to do the coalescing/cleanup if you really needed the pages back at that point.

Upvotes: 3

Related Questions