CODEWITHSUNDEEP
c++memorymallocthrifttcmalloc
Logan
Logan

Reputation: 1

tcmalloc allocation failures when there is free memory

I am using libtcmalloc_minimal.so.4.5.3 on Ubuntu 20. I am seeing a lot of allocation failures in the log of the form: src/central_freelist.cc:333] tcmalloc: allocation failed 8192

However, there is 290GB of free memory on the system according to free and the program is only using 90GB when it starts reporting errors.

The server sets the "tcmalloc.max_total_thread_cache_bytes" property to be equal to the amount of memory on the system, so I don't think that's the problem. The cgroup restricts it to 95% of the memory on the system. Is there something else I'm missing that might be limiting it?

The application is an apache thrift server running with approximately 32K threads if that matters.

I expect the program to use close to 95% of available memory on the system before getting allocation errors and I'm stumped as to what might be going on. Can anyone help?

Upvotes: 0

Views: 948

Answers (1)

Logan
Logan

Reputation: 1

It turns out that tcmalloc has a hard coded limit on the size of its thread caches and if you try to set the max_total_thread_cache_bytes to more than 1G it will silently ignore you. I went in and changed this limit in thread_cache.cc function set_overall_thread_cache_size and re-compiled and everything seems to work now.

Upvotes: 0

Related Questions