CODEWITHSUNDEEP
cmallocvirtualmemory-address
hlitz
hlitz

Reputation: 655

Limit Virtual Memory space for malloc()

I have written my own my_malloc() function that manages its own physical memory. In my application I want to be able use both the libc malloc() as well as my own my_malloc() function. So I somehow need to partition the virtual address space, malloc should always assign a virtual address only if its from its dedicated pool, same thing with my_malloc(). I cannot limit heap size, I just need to guarantee that malloc() and my_malloc() never return the same/overlapping virtual addresses.

thanks!

Upvotes: 0

Views: 1524

Answers (4)

Crashworks
Crashworks

Reputation: 41482

Reserve a large block of virtual address space, and have that be the pool from which my_malloc() allocates. Once you have reserved a large contiguous region of memory from the OS, then subsequent calls to malloc() have to come from elsewhere.

For example, on Windows, you can use VirtualAlloc() to reserve a 256mb block of space. The memory won't actually be allocated until you "commit" it with a subsequent call, but it will reserve an address range (such as 0x4000000-0x5000000) which subsequent malloc() will not use. Then your my_malloc() can commit blocks out of this reserved range as requested, and subdivide them by whatever allocation scheme you've written.

I'm told the equivalent Linux call is mmap(). (edit: I previously said "kmalloc or vmalloc, depending on whether you need the memory to be physically contiguous or not," but those are kernel-level functions.)

We use this mechanism in our app to redirect all allocations of a certain size into our own custom pooled-block allocator for speed and efficiency. Among other things, it lets us reserve virtual pages in certain specific sizes that are more efficient for the CPU to handle.

Upvotes: 1

sarnold
sarnold

Reputation: 104110

If you add an mmap(2) call near the start of the program, you can allocate as much memory as you need with whatever addresses you need (see the hint, that's usually left NULL for the OS to determine) immediately; that will prevent malloc(3), or any other memory allocation routines, from getting those particular pages.

Don't worry about the memory usage; since modern systems are quite happy to overcommit, you'll only use a few hundred kilobytes more kernel space to handle page tables. Not too bad.

Upvotes: 0

Adam Liss
Adam Liss

Reputation: 48330

One possibility would be to have my_malloc() call malloc() at startup to pre-allocate a large pool of memory, then apportion that memory to its callers and manage it accordingly. However, a full implementation would need to handle garbage collection and defragmentation.

Another possibility would be to have my_malloc() call malloc() each time it needs to allocate memory and simply handle whatever "bookkeeping" aspects you're interested in, such as number of blocks allocated, number of blocks freed, largest outstanding block, total allocated memory, etc. This is by far the safer and more efficient mechanism, as you're passing all of the "hard" operations to malloc().

Upvotes: 1

Edmund
Edmund

Reputation: 10829

One answer is to make your my_malloc use memory allocated by malloc. Using big enough blocks would mostly acheive that; then within each block your version will maintain its own structures and allocate parts of it to callers.

It gets tricky because you can't rely on extending the entire available space for your version, as you can when you get memory from sbrk or similar. So your version would have to maintain several blocks.

Upvotes: 1

Related Questions