CODEWITHSUNDEEP
pythonfile
Goku241
Goku241

Reputation: 81

Single Line from file is too big?

In python, I'm reading a large file, and I want to add each line(after some modifications) to an empty list. I want to do this to only the first few lines, so I did:

X = []
for line in range(3):

    i = file.readline()
    m = str(i)
    X.append(m)

However, an error shows up, and says there is a MemoryError for the line i = file.readline().

What should I do? It is the same even if I make the range 1 (although I don't know how that affects the line, since it's inside the loop).

How do I not get the error code? I'm iterating, and I can't make it into a binary file because the file isn't just integers - there's decimals and non-numerical characters.

The txt file is 5 gigs.

Any ideas?

Upvotes: 5

Views: 391

Answers (1)

Danielle M.
Danielle M.

Reputation: 3662

filehandle.readline() breaks lines via the newline character (\n) - if your file has gigantic lines, or no new lines at all, you'll need to figure out a different way of chunking it.

Normally you might read the file in chunks and process those chunks one by one.

Can you figure out how you might break up the file? Could you, for example, only read 1024 bytes at a time, and work with that chunk?

If not, it's often easier to clean up the format of the file instead of designing a complicated reader.

Upvotes: 4

Related Questions