CODEWITHSUNDEEP
pythonpython-2.7input
MJP
MJP

Reputation: 5627

What is the best way to handle large inputs in python (100 k lines)?

I need to handle input of 100k lines (each line contains a string) and perform a function on each line. The function will return one result per string and should print it to the console. What is the best way of doing this?

My current attempt is:

strings = []
for i in xrange(int(input())):
    strings.append(raw_input())

More background: I want to solve a problem on Hackerrank. An input can look like this (powered by Hackerrank): https://hr-testcases.s3.amazonaws.com/4187/input02.txt?AWSAccessKeyId=AKIAINGOTNJCTGAUP7NA&Expires=1420719780&Signature=iSzA93z7GKVIcn4NvdqAbbCOfMs%3D&response-content-type=text%2Fplain

Upvotes: 0

Views: 1356

Answers (2)

markcial
markcial

Reputation: 9333

use the stdin stream, the stdin is like a file stream

import sys
for line in sys.stdin
  do_work(line)

Upvotes: 0

Burhan Khalid
Burhan Khalid

Reputation: 174662

You don't need to store the entire file in memory because you are calculating and printing results as you read the file.

As such, simply read the file line-by-line, do your calculations and print the results:

with open('large-file.txt') as the_file:
    for line in the_file:
       result = do_something_with(line)
       print(result)

Upvotes: 2

Related Questions