Descriptors as instance attributes in python

Question

To the question:

Why can't descriptors be instance attributes?

it has been answered that:

descriptor objects needs to live in the class, not in the instance

because that is the way that the __getattribute__ is implemented.

A simple example. Consider a descriptor:

class Prop(object):

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj._value * obj._multiplier

    def __set__(self, obj, value):
        if obj is None:
            return self
        obj._value = value

class Obj(object):

    val = Prop()

    def __init__(self):
        self._value = 1
        self._multiplier = 0

Consider the case in which each obj has multiple Prop: I would need to use unique names to identify the values and multipliers (Like here. Having a per instance descriptor object would allow to store the _multiplier (and the _value) in the descriptor itself, simplifying a few things.

To implement per instance descriptor attributes you need to either:

1. create a per instance class See here
2. override __getattribute__ See here

I am aware that similar questions have been raised before, but I have not found a real explanation:

1. Why Python is designed this way?
2. What is the suggested way to store information that the descriptor needs but is per instance?

Answer 1

In Python 3.6 this can be done quite easily. Maybe it's not as intended but hey, if it works, right? ;)

Python 3.6 adds the __set_name__ method:

object.__set_name__(self, owner, name)

Called at the time the owning class owner is created. The descriptor has been assigned to name.

New in version 3.6.

Using this name to store the internal value in the instance's dict seems to work fine.

>>> class Prop:
...     def __set_name__(self, owner, name):
...         self.name = name
...     def __get__(self, instance, owner):
...         print('get')
...         return instance.__dict__.setdefault(self.name, None)
...     def __set__(self, instance, value):
...         print('set')
...         instance.__dict__[self.name] = value
... 
>>> class A:
...     prop = Prop()
... 
>>> a = A()
>>> a.prop = 'spam'
set
>>> a.prop
get
'spam'

Note that this is not a full descriptor implementation and of course if you decide to use it it's at your own risk.

Answer 2

This exact question was raised on Python-list earlier this year. I'm just going to quote Ian G. Kelly's response:

The behavior is by design. First, keeping object behavior in the class definition simplifies the implementation and also makes instance checks more meaningful. To borrow your Register example, if the "M" descriptor is defined by some instances rather than by the class, then knowing that the object "reg" is an instance of Register does not tell me anything about whether "reg.M" is a valid attribute or an error. As a result, I'll need to guard virtually every access of "reg.M" with a try-except construct just in case "reg" is the wrong kind of register.

Second, the separation of class from instance also helps you keep object behavior separate from object data. Consider the following class:

class ObjectHolder(object):
    def __init__(self, obj):
        self.obj = obj

Don't worry about what this class might be useful for. Just know that it's meant to hold and provide unrestricted access to arbitrary Python objects:

>>> holder = ObjectHolder(42)
>>> print(holder.obj) 42
>>> holder.obj = range(5)
>>> print(holder.obj) [0, 1, 2, 3, 4]

Since the class is meant to hold arbitrary objects, it's even valid that somebody might want to store a descriptor object there:

>>> holder.obj = property(lambda x: x.foo)
>>> print(holder.obj) <property object at 0x02415AE0>

Now suppose that Python invoked the descriptor protocol for descriptors stored in instance attributes:

>>> holder = ObjectHolder(None)
>>> holder.obj = property(lambda x: x.foo)
>>> print(holder.obj)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'ObjectHolder' object has no attribute 'foo'

In this case, the ObjectHolder would fail to simply hold the property object as data. The mere act of assigning the property object, a descriptor, to an instance attribute would change the behavior of the ObjectHolder. Instead of treating "holder.obj" as a simple data attribute, it would start invoking the descriptor protocol on accesses to "holder.obj" and ultimately redirect them to the non-existent and meaningless "holder.foo" attribute, which is certainly not what the author of the class intended.

If you want to be able to support multiple instances of a descriptor, just make that descriptor's constructor take a name argument (prefix), and prefix the added attributes with that name. You could even create a namespace object (dictionary) within the class instance to hold all of the new property instances.

Answer 3

Plenty of advanced functionality only works when defined on a class rather than an instance; all of the special methods, for example. As well as making code evaluation more efficient, this makes clear the separation between instances and types which otherwise would tend to collapse (because of course all types are objects).

I'm not sure how recommended this is, but you could in the instance store a mapping from descriptor instance to attribute value:

class Prop(object):
     def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj._value * obj._multiplier[self]

    def __set__(self, obj, value):
        if obj is None:
            return self
        obj._value = value

class Obj(object):
    val = Prop()

    def __init__(self):
        self._value = 1
        self._multiplier = {Obj.val: 0}

This has obvious advantages over the other two suggested options:

1. per-instance classes break object orientation and increase memory usage;
2. overriding __getattribute__ is inefficient (as all attribute access must go through the overridden special method) and is fragile.

---

As an alternative, you could use a proxy property:

class PerInstancePropertyProxy(object):
    def __init__(self, prop):
        self.prop = prop
    def __get__(self, instance, owner):
        if instance is None:
            return self
        return instance.__dict__[self.prop].__get__(instance, owner)
    def __set__(self, instance, value):
        instance.__dict__[self.prop].__set__(instance, value)
class Prop(object):
    def __init__(self, value, multiplier):
        self.value = value
        self.multiplier = multiplier
    def __get__(self, instance, owner):
        if instance is None:
            return self
        return self.value * self.multiplier
    def __set__(self, instance, value):
        self.value = value
class Obj(object):
    val = PerInstancePropertyProxy('val')
    def __init__(self):
        self.__dict__['val'] = Prop(1.0, 10.0)
    def prop(self, attr_name):
        return self.__dict__[attr_name]