Python: How do I dynamically alter methods of dict and list objects?

Question

Here is a mockup of what I want to do:

alist = [1,2,3,4,5]   # create a vanilla python list object
replacef (alist)      # replace __setitem__, extend,... with custom functions
alist[0]=2            # now the custom __setitem__ is called

This is for a DSL project where the syntax should be as close to normal python as possible, so subclassing list and making the user call something like alist = MyList(1,2,3,4,5) is not desirable. Also, because the DSL needs to coexist with other libraries, globally changing list and dict is not an option...

I have tried creating instancemethods and setting them directly on the object like alist.append = myFunc, but Python says those properties are read only. Changing the __class__ attribute seems to be not permitted as well.

Is what I am trying to do even possible in Python?

Update: here are some findings on what is possible with a subclass of object:

Given:

class y(object):
    def __init__(self,a,b,c):
    self.a = a
    self.b = b
    self.c = c
def f(self):
    print self
    print self.a
    print self.b
    print self.c

class x(object):
def __init__(self,a,b,c):
    self.a = a
    self.b = b
    self.c = c
def f(self):
    print "x.f()"

>>> objy = y(1,2,3)
>>> objx = x(4,5,6)
>>> objy.f()
  <__main__.y object at 0x02612650>
  1
  2
  3
>>> objx.f()
  x.f()

It is possible to change the class of objx on the fly like this:

>>> objx.__class__ = y
>>> objx.f()
<__main__.y object at 0x02612D90>
4
5
6

Also, one can dynamically add/change object methods on the fly, much like in javascript:

>>> def g(self,p):
       print self
       print p
>>> import new
>>> objy.g = new.instancemethod(g,objy,y)
>>> objy.g(42)
<__main__.y object at 0x02612650>
42

However, both approaches will fail with objects that are implemented in C like dict and list:

>>> def mypop(self):
        print "mypop"
        list.mypop(self)


>>> alist.pop = new.instancemethod(mypop,alist,list) 

AttributeError: 'list' object attribute 'pop' is read-only

>>> x = [1,2,3,4,5]
>>> x.__class__ = mylist

TypeError: __class__ assignment: only for heap types

The suggested approach to use something like alist = replacef(alist) would not work here, because replacef could be called from a __setattribute__ call like this:

alist = [1,2,3,4,5]
aDSLObject.items = alist #  __setattribute__ calls replacef on alist
alist[0] = ....

So while it indeed is possible to change object methods dynamically, it seems like it is not possible to alter the behaviour of objects implemented in C - or am I missing something?

Answer 1

"Explicit is better than implicit." You should just document that your users need to use MyList() instead of list(). You and your users will be happier in the long run.

Ruby allows what you want, by the way. With Ruby, you can open up global objects and just add new behaviors. I find this scary, and Ruby runs slower than Python, so I'm not actually urging you to switch to Ruby.

So, provide MyList(), and provide a way to take an existing list and wrap it in a MyList().

You could write MyList() to just take a bunch of arguments, and make a list very conveniently:

class MyList(object):
    def __init__(self, *args):
        self.lst = args
    # add other methods to MyList() here

then you could call this with MyList(1, 2, 3, 4) and the instance would have a member called lst with the value: [1, 2, 3, 4]

But now, if you want to make a MyList instance out of a list, how do you do that? If you do this:

new_list = [1, 3, 5]
x = MyList(new_list)

then you have just set x to: [[1, 3, 5]]

So I recommend you make MyList() just take a list argument and save it:

def ensure_list(seq):
    try:
        seq[0]
        return seq
    except TypeError:
        return list(seq)

class MyList(object):
    def __init__(self, seq):
        self.lst = ensure_list(seq)
    # add other methods to MyList() here

I just edited this answer. Originally, I had a call to list() where I now have put a call to ensure_list(). ensure_list() checks to see if you can index its argument; if you can, it is either a list already or something that can act like a list, and it is returned unchanged. If you can't index it, then ensure_list() calls list(seq) to try to turn it into a list. This will work with iterators and generators, forcing them to expand out to a list.

Answer 2

Maybe you can do the "alist = MyList(1,2,3,4,5)" thing inside the replaceref function?

def replacef(l):
    return MyList(l) # Return instance of list subclass that has custom __setitem__

alist = [1,2,3,4,5]
alist = replaceref(alist)

This way user would still use the standard list syntax when defining a list but would get the new setitem after replaceref call.