python scope issue with anonymous lambda in metaclass

Question

I am using a metaclass to define read-only properties (accessor methods) for a class, by adding a property with only a getter (a lambda) for each field declared by the class. I am finding different behavior depending on where I define the lambda. It works if I define the getter lambda in an external function that is called by the __new__ method of the metaclass, and not if I define the lambda in directly in the __new__ method of the metaclass.

def _getter(key):
    meth =  lambda self : self.__dict__[key]
    print "_getter: created lambda %s for key %s" % (meth, key)
    return meth


class ReadOnlyAccessors(type):

    def __new__(cls, clsName, bases, dict):

        for fname in dict.get('_fields',[]):
            key = "_%s" % fname

            # the way that works
            dict[fname] = property(_getter(key)) 

            # the way that doesn't
            # meth = lambda self : self.__dict__[key]
            # print "ReadOnlyAccessors.__new__: created lambda %s for key %s" % (meth, key)
            # dict[fname] = property(meth)

        return type.__new__(cls, clsName, bases, dict)


class ROThingy(object):
    __metaclass__ = ReadOnlyAccessors

    _fields = ("name", "number")

    def __init__(self, **initializers):
        for fname in self._fields:
            self.__dict__[ "_%s" % fname ] = initializers.get(fname, None)
        print self.__dict__


if __name__ == "__main__":
    rot = ROThingy(name="Fred", number=100)
    print "name = %s\nnumber = %d\n" % (rot.name, rot.number)

As currently written, execution looks like this:

[slass@zax src]$ python ReadOnlyAccessors.py
_getter: created lambda <function <lambda> at 0x7f652a4d88c0> for key _name
_getter: created lambda <function <lambda> at 0x7f652a4d8a28> for key _number
{'_number': 100, '_name': 'Fred'}
name = Fred
number = 100

Commenting out the line that follows "the way that works" and uncommenting the three lines following "the way that doesn't" produces this:

    [slass@zax src]$ python ReadOnlyAccessors.py
    ReadOnlyAccessors.__new__: created lambda <function <lambda> at 0x7f40f5db1938> for key _name
    ReadOnlyAccessors.__new__: created lambda <function <lambda> at 0x7f40f5db1aa0> for key _number
    {'_number': 100, '_name': 'Fred'}
    name = 100
    number = 100

Note that even though the rot.__dict__ shows that _name is 'Fred', the value returned by the name Property is 100.

Clearly I'm not understanding something about the scope in which I'm creating the lambdas.

I've been reading Guido's document about metaclass for accessors here: https://www.python.org/download/releases/2.2.3/descrintro/#cooperation as well as the Python docs for the Python Data Model and this http://code.activestate.com/recipes/307969-generating-getset-methods-using-a-metaclass/ recipe for creating accessors using a metaclass, and finally everything on StackOverflow that I can find, but I'm just not getting it.

Thank you.

-Mike

Answer 1

This is yet another manifestation of the "late-binding" of python closures and has nothing to do with metaclasses ;-) -- Although maybe the metaclass in play makes the actual issue harder to see... consider:

funcs = [lambda: x for x in range(30)]
print funcs[0]()  # 29!

The reason is that the lambda function looks up the value from the closure when it is called, not when it was created. In this case, even though i was 0 when the first function was created, when it gets called, i has a value of 29.

Now in your case, you've got the same thing happening, only with the variable key. One simple way to fix it is to bind the value as a keyword argument to the function (as those get evaluated at creation time):

funcs = [lambda _x=x: _x for x in range(30)]

or, in your case:

meth = lambda self, _key: self.__dict__[_key]

Answer 2

The problem has to do with scope. When you define meth with

meth = lambda self : self.__dict__[key]

the key variable is not a variable in meth's local scope. So when the meth function is called, key must be searched for in the enclosing scope. (See the LEGB rule.) It finds it in the scope of the __new__ method. However, by the time meth gets called, the value of key is not necessarily the value of key when meth was defined. Rather the value of key is the last value it was assigned to due to the for-loop. It happens to always be '_number'. So no matter what meth you call, the value of self.__dict__['_number'] is being returned.

You can see that this is what is happening by defining meth this way inside __new__:

    for fname in dict.get('_fields',[]):
        key = "_%s" % fname

        def meth(self):
            print(key) # See what `meth` believes `key` is
            return self.__dict__[key]

yields

_number    # key is always `_number`
_number
name = 100
number = 100

---

The reason why _getter works is because key gets passed to _getter. So when meth gets called, it finds the value of key in _getter's scope, where key retains the value it got when _getter was called.

---

If you want to use a lambda instead of _getter, you can do so by using a default value for key:

meth = lambda self, key=key: self.__dict__[key]

Now, inside meth, key is a local variable. So when meth is called, the value of key will be the value of key in the local scope. The default value is bound to the function at definition-time, so the right value is being bound to each meth lambda function.

Answer 3

key word is dynamic scope.

it's very easy to step into the trap here.

to make the problem easier, forget about OO, just try to think about following code:

arr = []
for i in range(5):
    arr.append(lambda: i)

for lmb in arr:
    print lmb()

and this code:

def lmb_gen(val):
    return lambda: val

arr = []
for i in range(5):
    arr.append(lmb_gen(i))

for lmb in arr:
    print lmb()

easy answer is the i in lambda is binded to the i in the for loop, which is keep changing before the lambda get called. that's why 5 4's get printed.

while, in the second examples, the val in lambda is binded to the parameter val, which varies every time when the lmb_gen get called. therefore, in another word, the environment is different.

the rule is, when a variable do not defined in a function, the variable is actually binded to the first one of the "outer" environments.

this phenomenon does not only happen in lambda's case, but happen in named functions' case.