CODEWITHSUNDEEP
python
Daniel Standage
Daniel Standage

Reputation: 8314

Iterating over chained file handles

I have a list of open (for reading) file handles. I would like to iterate over every line of every file using a single loop, rather than creating an outer loop to iterate over the list of file handles and a nested inner loop to iterate over the lines. It looks like itertools.chain() is designed for precisely this purpose, but I cannot get it to work. What am I doing wrong?

Here is a simple example to illustrate.

$ python
Python 2.7.10 (default, May 26 2015, 13:08:10) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> fh1 = open('file1', 'r')
>>> fh2 = open('file2', 'r')
>>> import itertools
>>> input = itertools.chain([fh1, fh2])
>>> for line in input:
...     print line
... 
<open file 'file1', mode 'r' at 0x10c2a6780>
<open file 'file2', mode 'r' at 0x10c2a64b0>

Upvotes: 3

Views: 479

Answers (2)

John Kugelman
John Kugelman

Reputation: 361927

Don't wrap the arguments in a list. Pass them directly.

input = itertools.chain(fh1, fh2)

If you already have a list you can use one of these:

files = [...]

input = itertools.chain(*files)
input = itertools.chain.from_iterable(files)

Upvotes: 4

Kasravnd
Kasravnd

Reputation: 107347

The itertools.chain will chain the iterable objects that you have passed to it and in this case you just passed one list ([fh1, fh2]) so it will just return an iterator contain that list.

If you want to chain your file objects you need to pass them outside the list or unpack the list :

input = itertools.chain(fh1, fh2)

Or:

input = itertools.chain(*[fh1, fh2])

Demo :

>>> list(chain(range(5),range(2),range(3)))
[0, 1, 2, 3, 4, 0, 1, 0, 1, 2]

Upvotes: 3

Related Questions