CODEWITHSUNDEEP
pythonpython-3.x
Gordon Bean
Gordon Bean

Reputation: 4612

Magic methods, operators, and getattribute - more to the story?

Two related questions. First the context:

From the python documentation:

These are the so-called “rich comparison” methods. The correspondence between operator symbols and method names is as follows: xlt(y), x<=y calls x.le(y), x==y calls x.eq(y), x!=y calls x.ne(y), x>y calls x.gt(y), and x>=y calls x.ge(y).

Consider the following:

class Foo:
    def __getattribute__(self, attr):
        print(attr)
        return super(Foo, self).__getattribute__(attr)

foo = Foo()

I would expect that ANY function called on foo should be printed, correct? From the documentation, I would expect that if I execute

foo < 1

then this should equate to

foo.__lt__(1)

which in turn should call foo.__getattribute__('__lt__'), correct?

But it doesn't. What I see instead are two requests for the __class__ attribute:

In [135]: foo < 1
__class__
__class__
Traceback (most recent call last):

  File "<ipython-input-135-ee30676e9187>", line 1, in <module>
    foo < 1

TypeError: unorderable types: Foo() < int()

If I exectute foo.__lt__(1) I get the behavior I expect:

In [137]: foo.__lt__(1)
__lt__
Out[137]: NotImplemented

1) Can anyone please explain?

I would like to write a class that captures ALL attribute requests (including function calls) and modifies the request slightly before performing them. In essence, I would like to write a class the uniformly modifies all methods (inherited or non-existent), but I don't want to re-write an overload for every method. My thought was to intercept the method call, do my business, and then proceed.

2) Can this be done?

Upvotes: 4

Views: 804

Answers (1)

Gordon Bean
Gordon Bean

Reputation: 4612

The link provided by ekhumoro has the answers to my questions. For the sake of completeness I'll summarize them here. From the docs:

For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.

While this doesn't fully explain my specific scenario (where including the special method in the class declaration will enable implicit functionality), it does introduce the idea that implicit invocations of special methods are tied to an objects metaclass (i.e. type). This understanding helps to understand the implications of this next snippet:

Implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass.

So, implicit invocation of special methods do not utilize the __getattribute__ method on either the class or the metaclass. In other words, you can't intercept implicit special method lookup implicitly.

Reiterating this conclusion, the documentation states:

Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).

So, the solution is that the special methods must be explicitly defined on the class object in order to be implicitly invoked.

In my example above, I explicitly defined the special method __lt__ on the class object, so the implicit invocation via the < operator worked. However, as mentioned, neither __getattribute__ method (on the class or metaclass) was called in the process because of the special optimization made within the interpreter.

Finally, if I want to create the desired behavior, I need to explicitly declare all supported special methods on my class object. There aren't that many of them, but there isn't a simple "catch-all" solution.

EDIT: For those waiting for the conclusion of this story, I found a nice compromise by programmatically defining the special methods on the class object before instantiating it. The following code snippet shows how a call to the pw function return a special object that intercepts the specified operators and returns a container with the operator performed on it's members:

def pw(container):
    '''Return a PairWise object, which intercepts operations and performs them 
    on the associated container.'''

    # Define the supported functions
    # Functions that must return a specific type (i.e. __str__) 
    #  cannot be implemented
    funs = ['__lt__','__le__','__eq__','__ne__','__gt__','__ge__']
    for f in ['add','sub','mul','truediv', 'floordiv', 
              'mod', 'divmod', 'pow', 'lshift', 'rshift',
              'and','or','xor']:
        funs.extend(['__{}__'.format(f), '__r{}__'.format(f)])
    for f in ['neg','pos','abs','invert']:
        funs.append('__{}__'.format(f))

    # Define the pairwise execution wrapper             
    def pairwise(fun, cont):
        def fun2(self, *args, **kwargs):
            t = type(cont)
            return t(type(c).__getattribute__(c, fun)(*args, **kwargs) \
                              for c in cont)
        return fun2

    # Define the PairWise class from the type constructor
    # i.e. type(name, bases, dict)
    PW = type('PW', (object,), 
              {fun: pairwise(fun, container) for fun in funs})

    # Return an instance of the pairwise class              
    return PW()

I can now execute pairwise operations on containters (lists, sets, tuples) with a syntactically simple call to pw:

In [266]: pw([1,2,3,4]) + 1
Out[266]: [2, 3, 4, 5]

In [267]: -pw({7,8,9,10})
Out[267]: {-10, -9, -8, -7}

Upvotes: 1

Related Questions