Track changes to lists and dictionaries in python?

Question

I have a class where the instances of this class needs to track the changes to its attributes.

Example: obj.att = 2 would be something that's easily trackable by simply overriding the __setattr__ of obj.

However, there is a problem when the attribute I want to change is an object itself, like a list or a dict.

How would I be able to track things like obj.att.append(1) or obj.att.pop(2)?

I'm thinking of extending the list or the dictionary class, but monkey patching instances of those classes once the obj and the obj.att is both initialized so that obj gets notified when things like .append is called. Somehow, that doesn't feel very elegant.

The other way that I can think of would be passing an instance of obj into the list initialization, but that would break a lot of existing code plus it seems even less elegant than the previous method.

Any other ideas/suggestions? Is there a simple solution that I'm missing here?

Answer 1

You can also wrap the dictionary or list methods you want to track and make sure you do what you want inside the wrapper. Here is a dict example:

from functools import wraps

def _setChanged(func):
    @wraps(func)
    def wrappedFunc(self, *args, **kwargs):
        self.changed = True
        return func(self, *args, **kwargs)
    return wrappedFunc

def _trackObjectMethods(calssToTrack):
    for methodName in dir(calssToTrack):
        if methodName in calssToTrack._destructive:
            setattr(calssToTrack, methodName, _setChanged(getattr(calssToTrack, methodName)))

class Dictionary(dict):
    _destructive = ('__delitem__', '__setitem__', 'clear', 'pop', 'popitem', 'setdefault', 'update')

    def __init__(self, *args, **kwargs):
        self.changed = False
        super().__init__(*args, **kwargs)

_trackObjectMethods(Dictionary)

d = Dictionary()
print(d.changed)
d["1"] = 'test'
print(d.changed)
d.changed = False
print(d.changed)
d["12"] = 'test2'
print(d.changed)

As you can see if any items of the dictionary changes, the custom variable that I added to my custom Dictionary object will be set to True. This way I can tell if the object has changes since the last time I had set the changed variable to False.

Answer 2

My jsonfile module detects changes of (nested) JSON compatible Python objects. Just subclass JSONFileRoot to adapt change detection for your needs.

>>> import jsonfile
>>> class Notify(jsonfile.JSONFileRoot):
...   def on_change(self):
...     print(f'notify: {self.data}')
... 
>>> test = Notify()
>>> test.data = 1
notify: 1
>>> test.data = [1,2,3]
notify: [1, 2, 3]
>>> test.data[0] = 12
notify: [12, 2, 3]
>>> test.data[1] = {"a":"b"}
notify: [12, {'a': 'b'}, 3]
>>> test.data[1]["a"] = 20
notify: [12, {'a': 20}, 3]

Note that it goes on the proxy class way how e-satis advised, without supporting sets.

Answer 3

You could take advantage of the abstract base classes in the collections module, which dict and list implement. This gives you a standard library interface to code against with a short list of methods to override, __getitem__, __setitem__, __delitem__, insert. Wrap the attributes in a trackable adapter inside __getattribute__.

import collections

class Trackable(object):
    def __getattribute__(self, name):
        attr = object.__getattribute__(self, name)
        if isinstance(attr, collections.MutableSequence):
            attr = TrackableSequence(self, attr)
        if isinstance(attr, collections.MutableMapping):
            attr = TrackableMapping(self, attr)
        return attr

    def __setattr__(self, name, value):
        object.__setattr__(self, name, value)
        # add change tracking


class TrackableSequence(collections.MutableSequence):
    def __init__(self, tracker, trackee):
        self.tracker = tracker
        self.trackee = trackee

    # override all MutableSequence's abstract methods
    # override the the mutator abstract methods to include change tracking


class TrackableMapping(collections.MutableMapping):
    def __init__(self, tracker, trackee):
        self.tracker = tracker
        self.trackee = trackee

    # override all MutableMapping's abstract methods
    # override the the mutator abstract methods to include change tracking

Answer 4

Instead of monkey patching, you can create a proxy class:

• Make a proxy class that inherit from dict/list/set whatever
• Intercept attribute setting, and if the value is a dict/list/set, wrap it into the proxy class
• In proxy class __getattribute__, make sure the method is called on the wrapped type, but take care of tracking before doing so.

Pro:

• no class alteration

Con:

• you are limited to a number of types you know and expect

Answer 5

I was curious how this might be accomplished when I saw the question, here is the solution I came up with. Not as simple as I would like it to be but it may be useful. First, here is the behavior:

class Tracker(object):
    def __init__(self):
        self.lst = trackable_type('lst', self, list)
        self.dct = trackable_type('dct', self, dict)
        self.revisions = {'lst': [], 'dct': []}


>>> obj = Tracker()            # create an instance of Tracker
>>> obj.lst.append(1)          # make some changes to list attribute
>>> obj.lst.extend([2, 3])
>>> obj.lst.pop()
3
>>> obj.dct['a'] = 5           # make some changes to dict attribute
>>> obj.dct.update({'b': 3})
>>> del obj.dct['a']
>>> obj.revisions              # check out revision history
{'lst': [[1], [1, 2, 3], [1, 2]], 'dct': [{'a': 5}, {'a': 5, 'b': 3}, {'b': 3}]}

Now the trackable_type() function that makes all of this possible:

def trackable_type(name, obj, base):
    def func_logger(func):
        def wrapped(self, *args, **kwargs):
            before = base(self)
            result = func(self, *args, **kwargs)
            after = base(self)
            if before != after:
                obj.revisions[name].append(after)
            return result
        return wrapped

    methods = (type(list.append), type(list.__setitem__))
    skip = set(['__iter__', '__len__', '__getattribute__'])
    class TrackableMeta(type):
        def __new__(cls, name, bases, dct):
            for attr in dir(base):
                if attr not in skip:
                    func = getattr(base, attr)
                    if isinstance(func, methods):
                        dct[attr] = func_logger(func)
            return type.__new__(cls, name, bases, dct)

    class TrackableObject(base):
        __metaclass__ = TrackableMeta

    return TrackableObject()

This basically uses a metaclass to override every method of an object to add some revision logging if the object changes. This is not super thoroughly tested and I haven't tried any other object types besides list and dict, but it seems to work okay for those.