CODEWITHSUNDEEP
pythonfilegzippython-2.x
pythonRcpp
pythonRcpp

Reputation: 2146

Read multiple gzip files to 1 fileobject in python

i want to read multiple gzip file to 1 file object currently i am doing

import gzip 
a = gzip.open(path2zipfile1) 
for line in a.readline()
  #do some stuff

but i need to read from say 2 files 

a = gzip.open(path2zipfile1)  #read zip1
a = gzip.open(path2zipfile2, 'rU') #appending file object with contents of 2nd file 
for line in a.readlines()
  #this should give me contents from zip1 then zip2

unable to find the right mode to do so

Upvotes: 0

Views: 1480

Answers (1)

georgexsh
georgexsh

Reputation: 16624

use itertools.chain:

import itertools, gzip

files = ['path2zipfile1', 'path2zipfile2']
it = (gzip.open(f, 'rt') for f in files)

for line in itertools.chain.from_iterable(it):
    print(line)

another version without itertools:

def gen(files):
    for f in files:
        fo = gzip.open(f, 'rt')
        while True:
            line = fo.readline()
            if not line:
                break
            yield line

files = ['path2zipfile1', 'path2zipfile2']
for line in gen(files):
    print(line)

Upvotes: 4

Related Questions