How can "NameError: free variable 'var' referenced before assignment in enclosing scope" occur in real code?

Question

While I was hanging out in the Python chatroom, someone dropped in and reported the following exception:

NameError: free variable 'var' referenced before assignment in enclosing scope

I'd never seen that error message before, and the user provided only a small code fragment that couldn't have caused the error by itself, so off I went googling for information, and ... there doesn't seem to be much. While I was searching, the user reported their problem solved as a "whitespace issue", and then left the room.

After playing around a bit, I've only been able to reproduce the exception with toy code like this:

def multiplier(n):
    def multiply(x):
        return x * n
    del n
    return multiply

Which gives me:

>>> triple = multiplier(3)
>>> triple(5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in multiply
NameError: free variable 'n' referenced before assignment in enclosing scope

All well and good, but I'm having a hard time working out how this exception could occur in the wild, given that my example above is

1. Pretty stupid
2. Unlikely to happen by accident

... but obviously it does, given the report I mentioned at the start of this question.

So - how can this specific exception occur in real code?

Answer 1

I just got this error message while working on a Flask app.

I was trying to fix an error about the database. I had a lot of code setting up the app in an app.py file (not in any function), but then I saw in the docs that this setup code should be in a create_app() function, so at first I created create_app() function at the bottom of the file and moved the creation of the db instance into it along with a few other lines of code I'd had inside an if __name__ == '__main__' clause.

That didn't work, so I then moved the def create_app() line up to the top of the file and indented almost all of the code in the file (which included some event handlers). That caused the creation of the db object to be beneath its usage in some of the event handlers I had indented, which caused this error.

I fixed the error by moving the creation of the db object back up towards the top of the create_app() function, above its appearance in the event handlers.

Answer 2

Too late to answer but I think I can give some detailed information to this situation. It would help future readers to see what's going on here.

So the error message says:

NameError: free variable 'var' referenced before assignment in enclosing scope

When we talk about free variables, we're dealing with nested functions. Python has done some "magic" in order to give nested functions the ability to access the variables defined inside their parent scope. If we have:

def outer():
    foo = 10
    def inner():
        print(foo)
    return inner

outer()()  # 10

Normally we shouldn't have access to foo in inner function. Why ? because after calling and executing the body of the outer function, its namespace is destroyed. Basically any local variable defined inside the function is no longer available after the function terminates.

But we have access...

That magic happens with the help of the "Cell object":

“Cell” objects are used to implement variables referenced by multiple scopes. For each such variable, a cell object is created to store the value; the local variables of each stack frame that references the value contains a reference to the cells from outer scopes which also use that variable. When the value is accessed, the value contained in the cell is used instead of the cell object itself.

Just to see that hidden stored value in cells(we'll talk about __closure__ a bit later):

def outer():
    foo = 10
    def inner():
        print(foo)
    return inner

print(outer().__closure__[0].cell_contents)  # 10

How does it work?

In "compile" time,

when Python sees a function within another function, it takes note of the name of the variables referenced inside the nested function which are actually defined in the outer function. This information is stored in both functions' code objects. co_cellvars for outer function and co_freevars for inner function:

def outer():
    foo = 10
    def inner():
        print(foo)
    return inner

print(outer.__code__.co_cellvars)   # ('foo',)
print(outer().__code__.co_freevars) # ('foo',)

Now execution time..., (see the code)

When Python wants to execute the outer function, it creates a "cell object" for each variables (co_cellvars) that it has taken a note of.

Then as it goes through the lines, whenever it sees an assignment to such variables, it fills the corresponding cell object with that variable. (remember, "they" contain the actual values indirectly.)

When the execution reaches the line of creating the inner function, Python takes all the created cell objects and make a tuple out of them. This tuple is then assigned to the inner function's __closure__.

The point is when this tuple is created, some of the cells may not have value yet. They are empty(see the output)!...

At this point when you call the inner function those cells without value will raise that mentioned error!

def outer():
    foo = 10

    def inner():
        print(foo)
        try:
            print(boo)
        except NameError as e:
            print(e)

    # Take a look at inner's __closure__ cells
    print(inner.__closure__)
    # So one boo is empty! This raises error
    inner()

    # Now lets look at inner's __closure__ cells one more time (they're filled now)
    boo = 20
    print(inner.__closure__)
    # This works fine now
    inner()

outer()

output from Python 3.10:

(<cell at 0x7f14a5b62710: empty>, <cell at 0x7f14a5b62830: int object at 0x7f14a6f00210>)
10
free variable 'boo' referenced before assignment in enclosing scope
(<cell at 0x7f14a5b62710: int object at 0x7f14a6f00350>, <cell at 0x7f14a5b62830: int object at 0x7f14a6f00210>)
10
20

The error free variable 'boo' referenced before assignment in enclosing scope makes sense now.

Note: This error is reworded in Python 3.11 to:

cannot access free variable 'boo' where it is not associated with a value in enclosing scope

But the idea is the same.

---

If you look at the bytecode of the outer function, you'd see the steps I mentioned in the "execution time" section in action:

from dis import dis

def outer():
    foo = 10
    def inner():
        print(foo)
        print(boo)
    boo = 20
    return inner

dis(outer)

output from Python 3.11:

              0 MAKE_CELL                1 (boo)
              2 MAKE_CELL                2 (foo)

  3           4 RESUME                   0

  4           6 LOAD_CONST               1 (10)
              8 STORE_DEREF              2 (foo)

  5          10 LOAD_CLOSURE             1 (boo)
             12 LOAD_CLOSURE             2 (foo)
             14 BUILD_TUPLE              2
             16 LOAD_CONST               2 (<code object inner at 0x7fb6d4731a30, file "", line 5>)
             18 MAKE_FUNCTION            8 (closure)
             20 STORE_FAST               0 (inner)

  8          22 LOAD_CONST               3 (20)
             24 STORE_DEREF              1 (boo)

  9          26 LOAD_FAST                0 (inner)
             28 RETURN_VALUE

MAKE_CELL is new in Python3.11.
STORE_DEREF stores the value inside the cell object.

Answer 3

Think of a more complex function where n is bound depending on some condition, or not. You don't have to del the name in question, it also happens if the compiler sees an assignment, so the name is local, but the code path is not taken and the name gets never assigned anything. Another stupid example:

def f():
    def g(x):
        return x * n
    if False:
        n = 10
    return g