CODEWITHSUNDEEP
mmap
camelccc
camelccc

Reputation: 2992

overhead of reserving address space using mmap

I have a program that routinely uses massive arrays, where the memory is allocated using mmap

Does anyone know the typical overheads of allocating address space in large amounts before the memory is committed, either if allocating with MAP_NORESERVE or backing the space with a sparse file? It5 strikes me mmap can't be free since it must make page table entries for the allocated space. I want to have some idea of this overhead before implementing an algorithm I'm considering.

Obviously the answer is going to be platform dependent, im most interested in x64 linux, sparc solaris and sparc linux. I'm thinking that the availability of 1mb pages makes the overhead rather less on a sparc than x64.

Upvotes: 2

Views: 2513

Answers (1)

Naruil
Naruil

Reputation: 2310

The overhead of mmap depends on the way you use it. And it is typically negligible when you use it in an appropriate way.

In linux kernel, mmap operation can be divided into two parts:

  1. look for a free address range that can hold the mapping

  2. Create/enlarge vma struct in address space (mm_struct)

So allocate large amount of memory use mmap do not introduce more overhead than small ones.

So you should allocate memory as larger as possible in each time. (avoid mutiple times of small mmap)

And you may provide the start address explicitly (if possible). This could save some time in kernel in looking for an large enough free space.

If your application is an multi-threaded program. You should avoid concurrent calls to mmap. That is because the address space is protected by a reader-writer lock and the mmap always takes the writer lock. mmap latency will be orders of magnitude greater in this case.

Moreover, mmap only create the mapping but not the page table. Pages are allocated in the page fault handler when being touched. Page fault handler would take the reader lock that protects address space and can also affects mmap performance.

In this case, you should always try to reuse your large array instead of munmap it and mmap again. (Avoid pagefaults)

Upvotes: 4

Related Questions