CODEWITHSUNDEEP
pythonpython-2.7filewith-statement
Satarupa Guha
Satarupa Guha

Reputation: 1307

How can I open multiple files (number of files unknown beforehand) using "with open" statement?

I need to use a with open statement for opening the files, because I need to open a few hundred files together and merge them using K-way merge.

So at this point, I need very fast I/O that does not store the whole/huge portion of file in memory (because there are hundreds of files, each of ~10MB). I just need to read one line at a time for K-way merge. Reducing memory usage is my primary focus right now.

I learned that with open is the most efficient technique, but I cannot understand how to open all the files together in a single with open statement.

Upvotes: 4

Views: 3446

Answers (3)

ShadowRanger
ShadowRanger

Reputation: 155506

While not a solution for 2.7, I should note there is one good, correct solution for 3.3+, contextlib.ExitStack, which can be used to do this correctly (surprisingly difficult to get right when you roll your own) and nicely:

from contextlib import ExitStack

with open('source_dataset.txt') as src_file, ExitStack() as stack:
    files = [stack.enter_context(open(fname, 'w')) for fname in fname_list]
    # do stuff with src_file and the values in files ...
# src_file and all elements in stack cleaned up on block exit

Importantly, if any of the opens fails, all of the opens that succeeded prior to that point will be cleaned up deterministically; most naive solutions end up failing to clean up in that case, relying on the garbage collector at best, and in cases like lock acquisition where there is no object to collect, failing to ever release the lock.

Posted here since this question was marked as the "original" for a duplicate that didn't specify Python version.

Upvotes: 5

martineau
martineau

Reputation: 123501

It's fairly easy to write your own context manager to handle this by using the built-in contextmanager function decorator to define "a factory function for with statement context managers" as the documentation puts it. For example:

from contextlib import contextmanager

@contextmanager
def multi_file_manager(files, mode='rt'):
    """ Open multiple files and make sure they all get closed. """
    files = [open(file, mode) for file in files]
    yield files
    for file in files:
        file.close()


if __name__ == '__main__':

    filenames = 'file1', 'file2', 'file3'

    with multi_file_manager(filenames) as files:
        a = files[0].readline()
        b = files[2].readline()
            ...

If you don't know all the files ahead of time, it would be equally easy to create a context manager that supported adding them incrementally with the context. In the code below, a contextlib.ContextDecorator is used as the base class to simplify the implementation of a MultiFileManager class.

from contextlib import ContextDecorator

class MultiFileManager(ContextDecorator):
    def __init__(self, files=None):
        self.files = [] if files is None else files

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        for file in self.files:
            file.close()

    def __iadd__(self, other):
        """Add file to be closed when leaving context."""
        self.files.append(other)
        return self


if __name__ == '__main__':

    filenames = 'mfm_file1.txt', 'mfm_file2.txt', 'mfm_file3.txt'

    with MultiFileManager() as mfmgr:
        for count, filename in enumerate(filenames, start=1):
            file = open(filename, 'w')
            mfmgr += file  # Add file to be closed later.
            file.write(f'this is file {count}\n')

Upvotes: 5

Brian Schlenker
Brian Schlenker

Reputation: 5426

with open(...) as f: 
    # do stuff

translates roughly to 

f = open(...)
# do stuff
f.close()

In your case, I wouldn't use the with open syntax. If you have a list of filenames, then do something like this

filenames = os.listdir(file_directory)
open_files = map(open, filenames)
# do stuff
for f in open_files:
    f.close()

If you really want to use the with open syntax, you can make your own context manager that accepts a list of filenames

class MultipleFileManager(object):
    def __init__(self, files):
        self.files = files

    def __enter__(self):
        self.open_files = map(open, self.files)
        return self.open_files

    def __exit__(self):
        for f in self.open_files:
            f.close()

And then use it like this:

filenames = os.listdir(file_directory)
with MulitpleFileManager(filenames) as files:
    for f in files:
        # do stuff

The only advantage I see to using a context manager in this case is that you can't forget to close the files. But there is nothing wrong with manually closing the files. And remember, the os will reclaim its resources when your program exits anyway.

Upvotes: 2

Related Questions