CODEWITHSUNDEEP
python
wuhaochi
wuhaochi

Reputation: 419

MemoryError when read a file larger than 2G

a single file with size greater then 2G. I call open(f, "rb").read() MemoryError. I call open(f, "rb").read(1<<30) OK

How can I eliminate the 2G limit? I have enough memory -- 16G

Upvotes: 0

Views: 752

Answers (3)

Burhan Khalid
Burhan Khalid

Reputation: 174614

As @itaypk said in the comments, 32bit executables are limited to 2GB of accessible ram (2^31). In order to utilize additional RAM, you would need to run a 64bit version of Python.

Depending on what you need to do with the file, you may not need to read it entirely. You can step through it:

with open('huge_file.txt') as f:
    for line in f:
       print line # print one line at a time

The above loop will not exhaust all available memory on your system.

Upvotes: 2

Alex Chamberlain
Alex Chamberlain

Reputation: 4207

What about using Memory Mapped Files (mmap)? There's a good example in the documentation on python.org. It is adapted below.

with open(f, "rb") as fi:
    # memory-map the file, size 0 means whole file
    mm = mmap.mmap(fi.fileno(), 0)

    # Do stuff

    mm.close()

Upvotes: 4

Owen
Owen

Reputation: 1736

Even though your file may be only 2GB, the overhead of reading it in may result in far more than 2GB of memory used.

Upvotes: 0

Related Questions