CODEWITHSUNDEEP
pythonparsing
Cerin
Cerin

Reputation: 64739

How to read specific part of large file in Python

Given a large file (hundreds of MB) how would I use Python to quickly read the content between a specific start and end index within the file?

Essentially, I'm looking for a more efficient way of doing:

open(filename).read()[start_index:end_index]

Upvotes: 21

Views: 43358

Answers (2)

Will Leeney
Will Leeney

Reputation: 11

This is my solution with variable width encoding. My CSV file contains a dictionary where each row is a new item. 

def get_stuff(filename, count, start_index):
    with open(filename, 'r') as infile:
             reader = csv.reader(infile)
             num = 0 
             for idx, row in enumerate(reader):
                 if idx >= start_index-1:
                     if num >= count:
                         return
                 else:
                     yield row 
                     num += 1

Upvotes: 1

Dan Lecocq
Dan Lecocq

Reputation: 3483

You can seek into the file the file and then read a certain amount from there. Seek allows you to get to a specific offset within a file, and then you can limit your read to only the number of bytes in that range.

with open(filename) as fin:
    fin.seek(start_index)
    data = fin.read(end_index - start_index)

That will only read that data that you're looking for.

Upvotes: 35

Related Questions