CODEWITHSUNDEEP
c++windowsvisual-studio-2010
user1950329
user1950329

Reputation:

How to force C++ allocate memory from disk?

I have thousands lines of C++ codes which work well on small text files, but crashes on huge text files (such as 2 GB size). Crash reason: app eats up memory.

Is it possible to allocate memory from disk? Because in most case, hard disk space is much bigger than physical memory. If I can borrow some space from hard disk for my app and return them back after use, then my app has little chance to crash.

Here are my design thoughts:

  1. Create a temporary file for file mapping(CreateFileMapping, OpenFileMapping)
  2. Force my app to allocate memory from the temporary file
  3. Do some clean up work: CloseFileMapping and delete the temporary file

Because I have so much existing code, if my design thought is reasonable, I don't want to redesign my project.

I'm not sure if the design thought is possible to implement. Anybody can help me?

PS: I'm using Visual C++ 2010.

Upvotes: 5

Views: 3352

Answers (3)

CapelliC
CapelliC

Reputation: 60034

If you wrote your algorithms using C++ library, you can see this question and related answers.

I actually used memory mapped files to handle large data files, (indexing, sorting, lazy updating), and I found it neither easy nor performing.

Upvotes: 0

Balog Pal
Balog Pal

Reputation: 17183

If you're running out of address space, the first thing is to look around what consumes the memory and whether it is really necessary to have. You may discover a design change that allows using less, or just keep a some elements in files -- regular files not playing memory.

Another alternative to consider is to emigrate some code to a different process. Using COM it is not hard to start and maintain, and the server can be a 64 bit process dodging the virtual address limit. I'd expect no more slowdown than with the disk-based idea and it's definitely simpler have.

Oh yeah, we skipped the first thing: maybe converting the project to 64 bit would solve the problem.

If your problem is not with virtual space you just want to consume less than say 1G in any case, your only friend is redesign. For that case I'd not try to fake VM, just do as written in first bullet.

Upvotes: 0

SigTerm
SigTerm

Reputation: 26429

I have thousands lines of C++ codes which work well on small text files, but crashes on huge text files (such as 2 GB size). Crash reason: app eats up memory.

You're trying to load entire file into memory on 32bit system (or on 64bit system with 2GB of RAM, or in 32bit application running on 64bit system). On 32bit system 2 gigabyte text file will not fit into memory no matter what you do, because with 32bit addressing you can operate on 2 gigabytes of RAM max, even if it is backed by paging file.

Solutions.

  1. (easiest) Process file line-by-line, without loading entire file into memory.
  2. Create file mapping. Even with file mapping you will not be able to access entire file at once, but you will be able to map portions of it into memory.

Upvotes: 3

Related Questions