CODEWITHSUNDEEP
c++cmemory-managementmemory-mapped-files
Ahmad
Ahmad

Reputation: 251

Memory allocators

I want to make a virtual allocator using c++ on windows,, which allocate data on a file on the hard disk, to reduce physical memory usage when allocate large objects !.. I don't want to use system virtual memory with virtualAlloc.. . I want to create a file on the disk and use it to allocate the whole objects and then move the part or the object that I need to the RAM .

I tried to use Memory mapped file , but I faced some problems: I used the mapped file to allocate vector elements, but when I bake to delete any of them, the address of the element changed, also I can't find a method to map the object only when needed "in my test I mapped the whole file"!

Any resources or open source projects can help ???

Upvotes: 3

Views: 2176

Answers (5)

MSalters
MSalters

Reputation: 179981

Probably the best way to solve this is not to return regular pointers to large objects. Simply return small proxies. These proxy objects implement the full interface of the larger object. However, these proxy objects can deal with the raw data being either in RAM or on disk. The proxies implement a LRU mechanism amongst themselves to optimize RAM use. The caller never sees the address of these proxies change, nor does it get any pointers to raw data.

Upvotes: 0

Éric Malenfant
Éric Malenfant

Reputation: 14148

You may find inspiration from Boost.Interprocess, which provides support for memory mapped files, as well as allocators and containers over that memory.

More information about the allocator design can also be found at http://www.boost.org/doc/libs/1_37_0/doc/html/interprocess/architecture.html

Upvotes: 2

MSalters
MSalters

Reputation: 179981

Sorry, but you fail to understand how (virtual) memory works. One the one hand you state that "I want to make "custom memory allocator" but without take a large space from the memory" but on the other hand you're surprised that "the address of the element changed".

This is pretty much to be expected. To make sure that the address of a (logical) object doesn't change, you have to keep the memory represented by that address committed to the object. If you free the memory, it becomes available for reuse, and so does the address. And if the address is reused, you can't page back the object to that address.

Ultimately, the problem here it that addresses and memory are very, very deeply connected. Recycling memory means recycling addresses.

Upvotes: 1

D.Shawley
D.Shawley

Reputation: 59583

Google can help here. I implemented a custom STL allocator a number of years ago that used a shared memory store. The same techniques can be used to implement a disk-backed allocator. I would start by looking at this SourceForge project for inspiration.

Upvotes: 3

joe
joe

Reputation: 35097

From http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=213

The POSIX header includes memory mapping syscalls and data structures. Because this interface is more intuitive and simpler than that of Windows, I base my memory mapping example on the POSIX library.

The mmap() system call:

caddr_t mmap(caddress_t map_addr, 
       size_t length, 
       int protection, 
       int flags, 
       int fd, 
       off_t offset);

Let's examine what each parameter means.

In the following example, the program maps the first 4 KB of a file passed in command line into its memory and then reads int value from it:

#include <errno.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/types.h>

 int main(int argc, char *argv[])
 {
 int fd;
 void * pregion;
 if (fd= open(argv[1], O_RDONLY) <0)
 {
 perror("failed on open");
 return –1;
 }
 /*map first 4 kilobytes of fd*/
 pregion=mmap(NULL, 4096, PROT_READ,MAP_SHARED,fd,0);
 if (pregion==(caddr_t)-1)
 {
 perror("mmap failed")
 return –1;
 }
 close(fd); //close the physical file because we don't need it
 //access mapped memory; read the first int in the mapped file
 int val= *((int*) pregion);
}

To unmap a mapped region, use the munmap() function:

int munmap(caddr_t addr, int length);

addr is the address of the region being unmapped. length specifies how much of the memory should be unmapped (you may unmap a portion of a previously-mapped region). The following example unmaps the first kilobyte of the previously-mapped file. The remaining three KB still remain mapped to the process's RAM after this call:

munmap(pregion, 1024);

Upvotes: 0

Related Questions