CODEWITHSUNDEEP
pythonpython-2.7
Anurag Sharma
Anurag Sharma

Reputation: 5049

Break a text file into smaller chunks in python

I have a text file (in MBs) and I want to break it into chunks(in KBs). I am simulating the file transfer behaviour over a network. So far I was able to make chunks according to the number of lines(seprated by '\n') inputed by user like this 

def make_chunks(fname):
    ifile = file(fname,'rb')
    file_iter = iter(ifile)
    args = [file_iter] * 10 # No of lines you want to have in one chunk
    chunks = list(izip_longest(fillvalue = None, *args))

But the chunks are now of different sizes.How would I make chunks of equal size(say 4KB)

Upvotes: 0

Views: 878

Answers (1)

Jon Clements
Jon Clements

Reputation: 142256

You can chunk by actual bytesize:

def chunk(fname):
    with open(fname, 'rb') as fin:
        return list(iter(lambda: fin.read(4096), ''))

Note that you might as well yield each chunk instead of building a list, and let the caller decide if it wants to build a list instead.

for chunk in iter(lambda: fin.read(4096), ''):
    yield chunk

Upvotes: 6

Related Questions