CODEWITHSUNDEEP
python
APE
APE

Reputation: 73

in python: using a list of file paths with the 'with' keyword

If I want to work with two files I can write:

with open(fname1, 'r') as f1, open(fname2, 'r') as f2:
    # do stuff with f1 and f2

But what if I have a list of paths (say, from glob.glob)? Can I do something analogous in a list comprehension? I have in mind something like:

with [open(path, 'r') for path in paths_list] as flist:
    # do stuff with this list of open file objects

As written, this doesn't work.

Upvotes: 3

Views: 298

Answers (3)

Marcin
Marcin

Reputation: 49856

The object of a with statement must be a context manager. So, no, you can't do this with a list, but you might be able to do it with a custom container.

See: http://docs.python.org/2/library/contextlib.html

Or, for 3.3+ there's this: http://docs.python.org/dev/library/contextlib.html#contextlib.ExitStack (Note, as per arbarnert's answer, this can be used in 2.7, using contextlib2. See his answer for the link.)

The actual solution here is probably to If you're not going to use contextlib2 put the context manager in a loop:

for path in paths_list:
    with open(path, 'r') as f:
         #whatever
         pass

Edit: Obviously, the above will open your files one at a time. There are relatively few use cases where you need to keep an undetermined number of files open at once.

Edit: To have multiple files open at once, ExitStack is the solution you are looking for.

Upvotes: 8

abarnert
abarnert

Reputation: 365935

In 3.3+, ExitStack is definitely the answer; in fact, it's the first example given in the docs:

with ExitStack() as stack:
    files = [stack.enter_context(open(path) for path in path_list]
    for f in files:
        do_something(f)

Of course if your with body is really just a loop over files, there's no reason to do this—just put a with statement for each file inside the loop. (In fact, there's a good reason not to do this—why open a possibly unbounded number of file handles at once just to use them one at a time?) But presumably your real code needs to use multiple files at the same time.

In earlier versions, you can easily just borrow ExitStack from the 3.3 source. Backporting to 3.2 is trivial; for 2.7, you need to strip out (or rewrite, if you need it) the stuff that gets fancy with exception propagation to guarantee you the right exception contexts, but that's pretty easy.

However, an even better solution is probably to install contextlib2 off PyPI, which "provides backports of features in the latest version of the standard library’s contextlib module to earlier Python versions." Then you can just use contextlib2.ExitStack instead of contextlib.ExitStack. (In fact, contextlib2 had ExitStack, under its preliminary name ContextStack, before Python 3.3 did…)

But you can also easily build a closing_all context manager, similar to the stdlib's closing but for multiple things:

@contextlib.contextmanager
def closing_all(things):
    try:
        yield things
    finally:
        for thing in things:
            thing.close()

If you need to deal with things whose close method can raise, you need to be a little smarter—but with file objects, and most other types you'd using with closing, you don't need that.

The bigger problem is that if any open can raise an exception, it's hard to find any valid sequence you can actually pass as the things argument. But if that's not a problem, using it is even simpler than ExitStack:

with closing_all(open(path) for path in path_list) as files:
    for f in fs:
        do_something(f)

You could also build an opening_all(paths, mode='r') that does the opens for you and wraps them in a closing_all, but I don't think that adds much.

Of course if you need to do this often, the best answer is to build opening_all around ExitStack and not even bother with closing_all.

Upvotes: 6

jh314
jh314

Reputation: 27812

You can use fileinput:

import fileinput

for line in fileinput.input(fileList):
    ...

Upvotes: 2

Related Questions