CODEWITHSUNDEEP
c++text-filesfilestreambinaryfiles
Ofer Magen
Ofer Magen

Reputation: 962

Reading large files in c++

Suppose I have a .txt file the size of 9 GB. And I only want to read the n'th MB. (I know what is n). But my computer has only 4GB of ram so I can't load all the file at once. I need to access different n's multiple times. What's the best way to do this (I don't know if the standart ifstream is able of doing this stuff).

Upvotes: 1

Views: 1350

Answers (2)

user2743554
user2743554

Reputation: 543

Is your OS 64-bit? If so, try mmap().

On modern operating systems, it is possible to mmap (pronounced “em-map”) a file to a region of memory. When this is done, the file can be accessed just like an array in the program.

This is more efficient than read or write, as only the regions of the file that a program actually accesses are loaded. Accesses to not-yet-loaded parts of the mmapped region are handled in the same way as swapped out pages.

Since mmapped pages can be stored back to their file when physical memory is low, it is possible to mmap files orders of magnitude larger than both the physical memory and swap space. The only limit is address space. The theoretical limit is 4GB on a 32-bit machine - however, the actual limit will be smaller since some areas will be reserved for other purposes. If the LFS interface is used the file size on 32-bit systems is not limited to 2GB (offsets are signed which reduces the addressable area of 4GB by half); the full 64-bit are available.

Memory mapping only works on entire pages of memory. Thus, addresses for mapping must be page-aligned, and length values will be rounded up.

More info:

Upvotes: 0

John Zwinck
John Zwinck

Reputation: 249652

You want to "seek" in the file to a specified location. In C++ using ifstream you use seekg(): http://www.cplusplus.com/reference/istream/istream/seekg/

For example:

char data[1024*1024];
ifstream in("myfile.txt");
in.seekg(450 * 1024 * 1024, ios_base::beg); // skip 450 MB
if (in.read(data, sizeof(data)) {
    // use data
}

Upvotes: 2

Related Questions