CODEWITHSUNDEEP
pythonpython-3.xclasspickle
Dimitry
Dimitry

Reputation: 2358

pickle object with overloaded __new__

In my program I'm using class derived from class of immutable objects (namedtuple). I've chosen to do so, because tuples seemed to result in significantly smaller sizes of files on disk (I didn't check their loadability at first.) Also, there are several static constant object in use. Depending on input __new__ can abort the calculation and return one of the static objects instead.

I've overrided the __new__ method. The __new__ takes less arguments than there are fields and does non-trivial computations to obtain other fields' values. When I try to unpickle the objects of the class, the __new__ method gets called in a way it should not. The example below is stripped version of the problem.

Example:

import pickle

class testclass(tuple):
     def __new__(cls,x):
         assert type(x)==int, "Wrong input type to constructor"
         return super().__new__(cls,[x,x*2,x**2])

pickle.loads(pickle.dumps( testclass(4),protocol = -1))

Output:

AssertionError

How do I create tuple objects with custom constructor, that are also serializable?

Extended example. The initial calculations are time consuming and unreversible:

import pickle, time

class testclass(tuple):
     def __new__(cls,params):
         assert type(params)==dict, "Wrong input type to constructor"
         a = params['sec']
         p,g = 23, 5
         r = p**a % g
         time.sleep(params['wait'])
         return super().__new__(cls,[p,g,r])

pickle.loads(pickle.dumps( testclass({'sec':4,'wait':1}),protocol = -1))

Upvotes: 0

Views: 756

Answers (2)

Dimitry
Dimitry

Reputation: 2358

Well, in the end, the answer I've settled with was "not to pickle object with overloaded __new__". 

import pickle, time
class Testclass(tuple):
     @classmethod 
     def new(cls,params):
         a = params['sec']
         p,g = 23, 5
         r = p**a % g
         time.sleep(params['wait'])
         return cls([p,g,r])

pickle.loads(pickle.dumps( Testclass.new({'sec':4,'wait':1}),protocol = -1))

In my code, I could replace all calls to constructor with explicit function calls. The creation\unpickling then was left to default constructors that don't do much beyond setting the fields.

Upvotes: 0

martineau
martineau

Reputation: 123473

You can do it by defining a __getnewargs__() method for pickle to use:

import pickle


class Testclass(tuple):
    def __new__(cls, x):
        assert isinstance(x, int), "Wrong input type to constructor"
        return super().__new__(cls, (x, x*2, x**2))

    def __getnewargs__(self):
        return (self[0],)  # Just return first element of sequence.


pickle.loads(pickle.dumps(Testclass(4), protocol=-1))

Upvotes: 1

Related Questions