CODEWITHSUNDEEP
pythonstdinreadline
Martin
Martin

Reputation: 5307

Python: Read huge number of lines from stdin

I'm trying to read a huge amount of lines from standard input with python.

more hugefile.txt | python readstdin.py

The problem is that the program freezes as soon as i've read just a single line.

print sys.stdin.read(8)
exit(1)

This prints the first 8 bytes but then i expect it to terminate but it never does. I think it's not really just reading the first bytes but trying to read the whole file into memory.

Same problem with sys.stdin.readline()

What i really want to do is of course to read all the lines but with a buffer so i don't run out of memory.

I'm using python 2.6

Upvotes: 7

Views: 9180

Answers (2)

Gringo Suave
Gringo Suave

Reputation: 31880

This should work efficiently in a modern Python:

import sys

for line in sys.stdin:
    # do something...
    print line,

You can then run the script like this:

python readstdin.py < hugefile.txt

Upvotes: 11

sarnold
sarnold

Reputation: 104060

Back in the day, you had to use xreadlines to get efficient huge line-at-a-time IO -- and the docs now ask that you use for line in file.

Of course, this is of assistance only if you're actually working on the lines one at a time. If you're just reading big binary blobs to pass onto something else, then your other mechanism might be as efficient.

Upvotes: 2

Related Questions