CODEWITHSUNDEEP
pythoninheritance
Matthias Urlichs
Matthias Urlichs

Reputation: 2544

Is it possible to use `_getattr__`-generated methods in a subclass?

I have a class whose methods may or may not be auto-generated. I want to be able to call these methods from a subclass, but can't figure out how to do that.

Consider this code:

class Upgrader:
    max = 99

    def _upgrade_no_op(self, db):
        if self.version < Upgrader.max:
            self.version += 1

    def __getattribute__(self, key):
        try:
            return super().__getattribute__(key)    
        except AttributeError:
            if key.startswith("upgrade_v"):    
                nr = int(key[9:])
                if 0 <= nr < self.max:                   
                    return self._upgrade_no_op
            raise

class Foo(Upgrader):
    version = 1
    def upgrade_v2(self):
        # None of these work:

        # Upgrade.upgrade_v2(self)
        # super().upgrade_v2()
        # super(Foo,self).upgrade_v2()
        # x = Upgrade.__getattribute__(self, "upgrade_v2");  # bound to Foo.upgrade_v2
        breakpoint()

Foo().upgrade_v2()

In my real code there are one or two other classes between Foo and Upgrader, but you get the idea.

Is there a way to do this, preferably without writing my own version of super()?

Upvotes: 0

Views: 29

Answers (1)

ShadowRanger
ShadowRanger

Reputation: 155546

You have a few problems:

  1. return super().__getattribute__(key) looks for __getattribute__ on a superclass (typically object), but the actual lookup of key is a plain lookup, it's not bypassing classes in the MRO, it's just restarting the lookup from first class in the MRO (the one that self is actually an instance of). super() is magic once; it'll skip past earlier classes in the MRO to perform the lookup that one time, then it's done. This is what you want when the attribute exists and you're being called from outside the class's methods (Foo().update_v2()'s initial call is going through Updater.__getattribute__ to find Foo.update_v2 in the first place), but when Foo.update_v2 tries to invoke a "maybe doesn't exist" parent version of update_v2, even when you manage to invoke the parent __getattribute__ (e.g. by directly invoking Upgrader.__getattribute__(self, "upgrade_v2")()), it's just going to give you Foo.update_v2 again, leading to infinite recursion.

  2. super() bypasses a lot of the dynamic lookup machinery; it will find only things with the appropriate name attached to each class after the one invoking it in the MRO, but it won't try invoking __getattr__ or __getattribute__ on each of those classes as it goes (that would slow things down dramatically, and complicate the code significantly). It's already generally a bad idea to rely on __getattribute__ to provide what amounts to core class functionality; having it dynamically intercept all lookups to insert things into the middle of inheritance chains during lookup is at best some truly intense code smell.

    99.99%+ of the time, you don't want dynamic lookup handling at all, and in the rare cases you do, you almost always want __getattr__ (which is only invoked when the name can't be found in any other way), not __getattribute__ (which is invoked unconditionally).

    Neither special method works with super()-triggered lookups the way you want though, so if you truly needed something like this, you'd be stuck re-implementing what super() is doing manually, but adding in dynamic lookup special method support as well (working over the instance's MRO manually, looking for what you want from each class, then if it lacks it, manually invoking __getattr__/__getattribute__ to see if it can generate it, then moving on to the next class if that fails too). This is insane; do not try to do it.

I strongly suspect you have an XY problem here. What you're trying to do is inherently brittle, and only makes any kind of sense in a complicated inheritance hierarchy where super().upgrade_v2() in Foo.upgrade_v2 might actually find a useful function in some class that is inherited along with Foo by some hypothetical third class that involves a diamond inheritance pattern leading back to Upgrader. That's already complicated enough, and now you're adding __getattribute__ (which slows every use of the class instances, and has a ton of pitfalls of its own even without inheritance of any kind); it's a bad idea.

If, as in this case, you have a smallish fixed set of methods that should exist, just generate them up front, and avoid __getattribute__ entirely:

class Upgrader:
    max = 99

    def _upgrade_no_op(self):
        if self.version < Upgrader.max:
            self.version += 1

# Dynamically bind _upgrade_no_op to each name we intend to support on Upgrader
for i in range(Upgrader.max):
    setattr(Upgrader, f'upgrade_v{i}', Upgrader._upgrade_no_op)

class Foo(Upgrader):
    version = 1
    def upgrade_v2(self):
        # Upgrade.upgrade_v2(self)
        super().upgrade_v2()  # Works just fine

f = Foo()
print(f.version)  # See original version of 1 here
f.upgrade_v2()
print(f.version)  # See updated version of 2 here

Try it online!

You code will work, run dramatically faster (no Python level function calls involved in every attribute and method lookup), and it won't drive maintainers insane trying to figure out what you're doing.

Upvotes: 2

Related Questions