CODEWITHSUNDEEP
pythonfilememory
user2255757
user2255757

Reputation: 766

10+ gig file reading problems Python

3 Files are to be read line by line, hence:

with open(file) as f:
    for line in f:
        print line

or 

for line in open(file):
    print line

Tried both line by line readers, but as soon as the file sizes start exceeding 10 GB python chooses to try and read the whole file into memory anyway... (works fine for file sizes <10 GB)

Any idea why?

Upvotes: 0

Views: 134

Answers (1)

Roman Kutlak
Roman Kutlak

Reputation: 2784

You can use an optional parameter to limit how many characters you can read at a time:

with open(file, "r") as f:
    line = f.readline(max_chars)
    while line:
        print(line, end='')
        line = f.readline(max_chars)

Upvotes: 1

Related Questions