How is a python function's name reference found inside it's declaration?

Question

It's totally unexpected (to me at least) that foo would know foo inside of the function def for foo. What the heck is going on here?

>>> def foo(x):
...   print "wow"
...   print globals().get('foo', 'sorry')
...   return foo
... 
>>> f = foo(3)
wow
<function foo at 0x10135f8c0>
>>> f
<function foo at 0x10135f8c0>

Is this some sort of effect of python's lazy evaluation? It builds the function code first and puts it in globals, but then actually builds the function later when it's called? Wow... what form of python magic is this?

Of course, this makes for ease of recursion... and was probably the reason this is in the language...

>>> def bar(x):
...   return bar(x)
... 
>>> bar(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in bar
  File "<stdin>", line 2, in bar
  File "<stdin>", line 2, in bar
  ...snip...
  File "<stdin>", line 2, in bar
  File "<stdin>", line 2, in bar
RuntimeError: maximum recursion depth exceeded

Answer 1

Nothing really mysterious.

There is, I believe a two pass aspect to this and you are doing a late evaluation in the body of the function.

update: I qualified my explanation of what happens after the definition step - it looks as some things do happen with the function code before it gets called. Wally's answer is better than mine in that respect.

Pass #1 - def foo(x) is found and set on the module's namespace. Probably adds the function parameters as well.

That's all, you don't drill down in the code.

update: Actually, I am not so sure about the code not being processed in some way before the call. Looking at foo.func_code internals showed no great difference on before and after call.

Also, if you have a syntax error like parenthesis nesting or whitespace problems that does show up before even calling the function so something's picking that up ahead of the call. See my code at the end.

I'll guess that the code is parsed but any variable resolution gets deferred until actual execution.

Pass #2 - you call foo(x), it is found and called.

The function's code gets executed for the first time.

When you hit globals()["foo"], it picks up the existing reference stored in #1.

You can also see some hints of this behavior when you run coverage.py or similar. On import, all the outer definitions in a module are flagged as covered.

But the actual code only gets covered when you call it.

Another way to think of it is that you need that namespacing pass first to set up references, before proceeding further. Otherwise, in my code below, foo would not find bar.

Here's some code I played to distinguish execution error vs syntax error...

def foo(x):
    """comment/uncomment to see behavior"""
    pass
    # return bar(x)   #this works

    return bar2(x)  #call time error

    #   return bar(x) bad whitespace  #IndentationError, nothing runs

print "foo defined"

def bar(x):
    return x*2


print "calling foo#1"

try:
    print foo(3)
except Exception, e:
    print e

#let's make it so there is a bar2...
bar2 = bar

print "calling foo#2"

try:
    print foo(6)
except Exception, e:
    print e

Answer 2

Maybe dis can help...

>>> def foo(x):
...   print "wow"
...   print globals().get('foo', 'sorry')
...   return foo
... 
>>> import dis
>>> dis.dis(foo)
  2           0 LOAD_CONST               1 ('wow')
              3 PRINT_ITEM          
              4 PRINT_NEWLINE       

  3           5 LOAD_GLOBAL              0 (globals)
              8 CALL_FUNCTION            0
             11 LOAD_ATTR                1 (get)
             14 LOAD_CONST               2 ('foo')
             17 LOAD_CONST               3 ('sorry')
             20 CALL_FUNCTION            2
             23 PRINT_ITEM          
             24 PRINT_NEWLINE       

  4          25 LOAD_GLOBAL              2 (foo)
             28 RETURN_VALUE        
>>> 

Hmm. Complicated. So let's go simpler...

>>> def zap(x):
...   return zap
... 
>>> dis.dis(zap)
  2           0 LOAD_GLOBAL              0 (zap)
              3 RETURN_VALUE        
>>> 

Yeah it looks like the bytecode is built, and hold instructions to load zap from globals. So, the two-step process makes zap inside of zap not a special thing at all.

Let's see if we can dig into the process better and clarify...

>>> def blah(x):
...   def hlab(y):
...     return blah(x) 
...   return hlab
... 
>>> blah.func_code.co_consts
(None, <code object hlab at 0x10fcdfd30, file "<stdin>", line 2>)
>>> b = blah(4)
>>> b 
<function hlab at 0x10fce9c80>
>>> dis.dis(blah.func_code.co_consts[-1])
  3           0 LOAD_GLOBAL              0 (blah)
              3 LOAD_DEREF               0 (x)
              6 CALL_FUNCTION            1
              9 RETURN_VALUE        
>>> dis.dis(b)
  3           0 LOAD_GLOBAL              0 (blah)
              3 LOAD_DEREF               0 (x)
              6 CALL_FUNCTION            1
              9 RETURN_VALUE        
>>> b_ = blah.func_code.co_consts[-1]   
>>> b.func_code
<code object hlab at 0x10fcdfd30, file "<stdin>", line 2>
>>> b_
<code object hlab at 0x10fcdfd30, file "<stdin>", line 2>
>>> 

So it looks like the bytecode is built first, then the function is built from that... which then points back to the original bytecode. I still don't see how it's hooked up, but I assume that's done on the "stack" somehow. This process, at least, would make whatever name references used inside of the function def not special at all (i.e. irrelevant if foo uses foo or blah or whatever).

So ok, I get it. Nothing special.

Although this is a bit odd...

>>> b(2)
<function hlab at 0x10faf9410>
>>> b(2)(2)
<function hlab at 0x10faf9578>
>>> b(2)(2)(2)
<function hlab at 0x10faf9410>
>>> _ is b(2)
False

...but I assume it's just cycling available memory addresses, or something like that.