"with", context manager, python: What's going on in simple terms?

Question

Novice Python coder here coming from a Java background. I'm still puzzled by this:

with open(...) as f:
    do_something(f)

even after Googling and reading some of the answers here (I just couldn't get my head around them).

My understanding is that there is this thing called a context manager that is some sort of wrapper that contains a reference to a file that is created. Regarding

as f:

the 'as' above is like the 'as' below

import numpy as np

It's just an alias. 'f' doesn't refer to a file, but to the context manager. The context manager, using the decorator pattern, implements all the methods the file that is opened does, so that I can treat it like a file object (and get at the file object by calling the appropriate methods, which will be called on the file inside the context manager). And, of course, the file is closed when the block completes (the whole point of this).

This begs the question: Does open() in general return a file (or a reference to a file), or a context manager? Does it return context managers in general, and that's what we've been using all the time without knowing it? Or does it return file types except in this special context when returns something different like a context manager.

Is this anywhere near right? Would anyone like to clarify?

Answer 1

This is the most basic context manager you could create:

class UselessContextManager(object):
    def __enter__(self):
        pass

    def __exit__(self, type, value, traceback):
        pass

with UselessContextManager() as nothing:
    print(nothing is None)

If you want to get a little feel for what the actually process flow looks like, try this one:

class PrintingContextManager(object):                                          
    def __init__(self, *args, **kwargs):                                       
        print('Initializing with args: {} and kwargs: {}'.format(args, kwargs))

    def __enter__(self):                                                       
        print('I am entering the context')                                     
        print('I am returning 42')                                             
        return 42                                                              

    def __exit__(self, type, value, traceback):                                
        print('And now I am exiting')                                          


print('Creating manager')                                                      
manager = PrintingContextManager()                                             
print('Entering with block')                                                   
with manager as fnord:                                                         
    print('Fnord is {}'.format(fnord))                                         
    print('End of context')                                                    
print('Out of context')                                                        

Output:

Creating manager
Initializing with args: () and kwargs: {}
Entering with block
I am entering the context
I am returning 42
Fnord is 42
End of context
And now I am exiting
Out of context

You should try modifying the code to print out type, value, traceback and then raise an exception inside the with block.

As you can see, the with syntax is almost just short for:

thing = ContextManager()
try:
    stuff = thing.__enter__()
except Exception as e:
    stuff.__exit__(type(e), e.args[0], e.__traceback__)

Though truthfully it's a bit different

You can see that files are always context managers:

>>> f = open('/tmp/spanish_inquisition.txt', 'w')
>>> f.__enter__
<function TextIOWrapper.__enter__>
>>> f.__exit__
<function TextIOWrapper.__exit__>

I didn't know a File could be a ContextManager simply by implementing two methods, without inheriting from a super class or explicitly implementing an interface. Again, I'm new to this language.

In Python that is explicitly implementing an interface. In Java, you have to specify what interface you want to adhere to. In Python, you just do it. Need a file-like object? Add a .read() method. Maybe .seek(), .open(), and .close() depending on what they expect. But in Python...

it = DecoyDuck()
if it.walks_like_a_duck() and it.talks_like_a_duck() and it.quacks_like_a_duck():
    print('It must be a duck')

Answer 2

File objects are themselves context managers, in that they have __enter__ and __exit__ methods. with notifies the file object when the context is entered and exited (by calling __enter__ and __exit__, respectively), and this is how a file object "knows" to close the file. There is no wrapper object involved here; file objects provide those two methods (in Java terms you could say that file objects implement the context manager interface).

Note that as is not an alias just like import module as altname; instead, the return value of contextmanager.__enter__() is assigned to the target. The fileobject.__enter__() method returns self (so the file object itself), to make it easier to use the syntax:

with open(...) as fileobj:

If fileobject.__enter__() did not do this but either returned None or another object, you couldn't inline the open() call; to keep a reference to the returned file object you'd have to assign the result of open() to a variable first before using it as a context manager:

fileobj = open(...)
with fileobj as something_enter_returned:
    fileobj.write()

or

fileobj = open(...)
with fileobj:  # no as, ignore whatever fileobj.__enter__() produced 
    fileobj.write()

Note that nothing stops you from using the latter pattern in your own code; you don't have to use an as target part here if you already have another reference to the file object, or simply don't need to even access the file object further.

However, other context managers could return something different. Some database connectors return a database cursor:

conn = database.connect(....)
with conn as cursor:
    cursor.execute(...)

and exiting the context causes the transaction to be committed or rolled back (depending on wether or not there was an exception).

Answer 3

Context managers are fairly simple beasts ... They're simply classes that define two separate methods (__enter__ and __exit__). Whatever is returned from __enter__ is bound in the as x clause of the with statement when the with statement is executed.

Here's a really stupid example:

>>> class CM(object):
...   def __enter__(self):
...     print('In __enter__')
...     return 'Hello world'
...   def __exit__(self, *args):
...     print('In __exit__')
... 
>>> with CM() as x:
...   print(x)
... 
In __enter__
Hello world
In __exit__

You'll frequently see context managers simply returning self from the __enter__ method, but I wrote the example above to demonstrate that you don't have to. Also note that you don't need to construct the context manager in the with statement, you can construct it ahead of time:

cm = CM()
with cm as x:
    ...

The reason for context managers is that when used in conjunction with the with statement, python guarantees that __exit__ will be called (even if an exception happens inside the with suite)¹.

file objects are implemented using the context manager API (they have well defined __enter__ and __exit__ methods) so file objects are context managers. When used with a with statement, python guarantees that when the with suite is exited, the file will be closed.

^{1barring catastrophic system failure -- e.g. if your computer blows up...}