Dynamically reassign __len__ in a custom Python class

Question

It used to be possible to set internal functions like __len__() at runtime. Here is an example:

#! /usr/bin/python3

import sys

class FakeSequence:
    def __init__(self):
        self.real_sequence = list()
        self.append = self.real_sequence.append
        self.__len__ = self.real_sequence.__len__

    def workaround__len__(self):
        return len(self.real_sequence)

if __name__ == '__main__':
    fake_sequence = FakeSequence()
    fake_sequence.append(1)
    fake_sequence.append(2)
    fake_sequence.append(3)

    length = len(fake_sequence)
    sys.stdout.write("len(fake_sequence) is %d\n" % (length))

Here are the results when you try to run it:

$ python2 len_test
len(fake_sequence) is 3

$ python3 len_test
Traceback (most recent call last):
  File "len_test", line 18, in <module>
    length = len(fake_sequence)
TypeError: object of type 'FakeSequence' has no len()

If I define the __len__() method as part of the class (remove the 'workaround' above), it works as you would expect. If I define __len__() and reassign it as above FakeSequence.__len__() is called, it does not access the newly assigned __len__(), it always calls the FakeSequence class method.

Can you point me to documentation that would help explain why assigning instance methods for member functions no longer works? Note that assigning non-double-underscore methods still works fine. I can work around this easily enough, I'm more concerned that I missed something fundamental in the transition from Python 2 to Python 3. The behavior above is consistent with the Python 3 interpreters I have easy access to (3.4, 3.6, 3.7).

Answer 1

Magic methods are only looked up on classes, not on instances, as documented here. And it's also the case in Py2 for new-style classes (cf https://docs.python.org/2.7/reference/datamodel.html#special-method-lookup-for-new-style-classes).

I assume the main motivations is to cut down on lookups for better performances, but there might be other reasons, can't tell.

EDIT: actually, the motivations are clearly explained in the 2.7 doc:

The rationale behind this behaviour lies with a number of special methods such as hash() and repr() that are implemented by all objects, including type objects. If the implicit lookup of these methods used the conventional lookup process, they would fail when invoked on the type object itself:

Then:

Incorrectly attempting to invoke an unbound method of a class in this way is sometimes referred to as ‘metaclass confusion’, and is avoided by bypassing the instance when looking up special methods:

And finally:

In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the getattribute() method even of the object’s metaclass

Bypassing the getattribute() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods

So that's indeed mostly a performance optimization - which is not much of a surprise when you know about Python's attribute lookup mechanism and how Python's "methods" are implemented.

Answer 2

The __len__ function is a class attribute (search for "magic methods"). In Python3 you should derive your custom class from other (base-)classes, e.g. object (search for "new style classes").

So if len() should be called on your custom class, the most easy way is to inherit from list (which provides append(), too), and override the __len__ method.

import sys

class FakeSequence(list):
    def __init__(self, *args, **kwargs):
        # not really necessary, leave out __init__ method, if
        # you don't have own attributes in your class.
        # but if you define an __init__ method, you
        # MUST call baseclass.__init__ inside, preferably
        # on top of the __init__ method
        list.__init__(self, *args, **kwargs)

    def __len__(self, *args, **kwargs):
        len_of_fakesequence = list.__len__(self, *args, **kwargs)
        # here you can do anything about len()
        return len_of_fakesequence

if __name__ == '__main__':
    fake_sequence = FakeSequence()
    fake_sequence.append(1)
    fake_sequence.append(2)
    fake_sequence.append(3)

    length = len(fake_sequence)
    sys.stdout.write("len(fake_sequence) is %d\n" % (length))

For sure it's not necessary to inherit from list, new style classes may inherit from any other class, at least object. In this case you have nothing to override, so every method has to be defined explicitely.

class AnyList(object):
    def __init__(self, *args, **kwargs):
        self.mylength = 0

    def __len__(self, *args, **kwargs): # len()
        return self.mylength

    def append(self):
        self.mylength += 1

if __name__ == '__main__':
    fake_sequence = AnyList()
    fake_sequence.append()
    fake_sequence.append()

    print("len(AnyList) is %d" % len(fake_sequence))
    # reads out 2

For most precise information, read the Chapter "3. Data model", especially "3.3.7. Emulating container types" of the python documentation.

Answer 3

Tested in Python 3:

You can create your own function, say mylen, and pass it to the class's constructor. The below example uses a function mylen that always returns 5:

import sys

class FakeSequence:
    def __init__(self, length_function):
        self.real_sequence = list()
        self.append = self.real_sequence.append
        self.length_function = length_function

    def __len__(self):
        return self.length_function()

if __name__ == '__main__':

    def mylen():
        return 5

    fake_sequence = FakeSequence(mylen)
    fake_sequence.append(1)
    fake_sequence.append(2)
    fake_sequence.append(3)

    length = len(fake_sequence)
    sys.stdout.write("len(fake_sequence) is %d\n" % (length))

Answer 4

This behavior is described in the docs here. This has to do with new- and old- style classes in Python 2 and 3. In other words, this shouldn't work in Python 2 if you had inherited from object. The code you posted uses old-style classes in Python 2 and new-style classes in Python 3.

The docs state that, in the interest of speed optimizations by bypassing look-ups, "the special method must be set on the class object itself in order to be consistently invoked by the interpreter."

Answer 5

For me this still works in python3:

In [1]: class Foo: pass

In [2]: len(Foo())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-955ff12672b5> in <module>()
----> 1 len(Foo())

TypeError: object of type 'Foo' has no len()

In [3]: Foo.__len__ = lambda _: 123

In [4]: len(Foo())
Out[4]: 123