CODEWITHSUNDEEP
c++boostfreezememory-mapped-files
TheShadow
TheShadow

Reputation: 69

C++: Creating large memory mapped file freezes pc

I'm trying to create a large memory mapped file with data that I want to use in a project. I'm using Boost library for the mapped files and everything seems to run fine until I try to create a big file (>2GB). My computer only has 3GB of RAM, and during the creation of the file it freezes. Is there a way to create the file using small chunks of data and avoid the freezing of the process? 

char p[20] = "test.raw";
boost::iostreams::mapped_file_params  params;
params.path = "test.raw";
params.new_file_size = sizeof(struct Rectangle) * data_size;
params.mode = std::ios_base::out;
boost::iostreams::mapped_file  file;
file.open(params);
struct Rectangle* dataMAP = (struct Rectangle*) file.data();
for(unsigned long long i=0; i<data_size; i++){
    dataMAP[i] = struct Rectangle(rand_float_array);
}

I'm running the program in 64-bit mode and have enabled /LARGEADRESSAWARE.

UPDATE

I'm currently trying to run a small example using the "Boost Interprocess" library but I cannot get it to run. This is my code based on this answer:

std::string vecFile = "vector.dat";
bi::managed_mapped_file file_vec(bi::open_or_create,vecFile.c_str(), sizeof(struct Rectangle) * data_size);

typedef bi::allocator<struct Rectangle, bi::managed_mapped_file::segment_manager> rect_alloc;
typedef std::vector<struct Rectangle, rect_alloc>  MyVec;

MyVec * vecptr = file_vec.find_or_construct<MyVec>("myvector")(file_vec.get_segment_manager());

vecptr->push_back(random_rectangle);

But the compiler says that it could not deduce the template argument for '_Ty*' from 'boost::interprocess::offset_ptr'. What am I doing wrong?

Upvotes: 1

Views: 490

Answers (1)

Mark Waterman
Mark Waterman

Reputation: 961

boost::iostreams calls the Win32 CreateFileMapping function behind the scenes here ... this function gives you a view into the file. And note the zero values that boost is passing as dwMaximumSizeHigh and dwMaximumSizeLow arguments--it's asking for a mapped view into that entire huge file.

It looks like there are quite a few limitations with CreateFileMapping() listed here. Note the part that says "The size of a file view is limited to the largest available contiguous block of unreserved virtual memory." ...so I think iostreams is actually asking your system to allocate a huge chunk of virtual memory, which evidently puts your system into a death spiral.

Since you're already using Boost, I'd give its interprocess library a shot instead. It has good memory mapped file support, STL-like containers that can be mapped to an MMF, and it supports mapping smaller regions of that huge file.

Upvotes: 1

Related Questions