CODEWITHSUNDEEP
pythonmodulepickle
shaoyl85
shaoyl85

Reputation: 1964

Pickle dump an object referencing a module

I would like to pickle-dump an object like this:

import module_A
import module_B

class SomeClass(object):
  def __init__(self, the_module):
    self.module = the_module
    self.some_other_members = whatever

x = SomeClass(module_A)

It is not allowed and the error is that I cannot dump a module. In this case I'm not interested in actually dumping the module. However, which module the object is using matters. So I would like to dump some identifier of the module such that on __setstate__ I can point self.module to the right module. What kind of identifiers should I use such that on __setstate__ I can find the right module by that identifier? There may be many possible modules and the class SomeClass does now know which of them are there.

UPDATE:

Sorry the error message we found is when we were using cPickle. The ordinary pickle does not have this problem, and as Mike mentioned, dill supports pickling modules too. The final solution I'm using is the one given here: https://mail.python.org/pipermail/python-ideas/2013-July/021959.html.

Upvotes: 1

Views: 1140

Answers (2)

Mike McKerns
Mike McKerns

Reputation: 35217

I think you can do exactly what you want with dill, which provides better serialization than pickle. 

>>> import dill
>>> import numpy
>>>
>>> import math
>>> 
>>> class SomeClass(object):
...   def __init__(self, module):
...     self.module = module
...     self.other = lambda x: numpy.arange(3)
... 
>>> # save the class instance
>>> x = SomeClass(math)
>>> _x = dill.dumps(x)
>>>
>>> # delete a bunch of stuff
>>> del SomeClass 
>>> del math
>>> import sys
>>> del sys.modules['math']
>>> del x
>>>
>>> # unpickle
>>> x = dill.loads(_x)
>>> x.module
>>> x.module.sin(x.other(0)[0])
0.0

Upvotes: 1

JL Peyret
JL Peyret

Reputation: 12174

Well, hardly a total answer, but when I took a look at a module I set on a 'dummy', a test object, I get.

print dummy.module.__file__

output

'lib/utils.pyc'

and some inspection:

>>> type(dummy.module)
<type 'module'>
>>> type(dummy.module.__file__)
<type 'str'>
>>>

Maybe you can save "lib/utils" from the above (during your __getstate__) and then do the appropriate import incantations to get your module back during __setstate__?

You dont have to bother with import at all, if the module is already loaded:

dummy.module2 = sys.modules["lib.utils"]

Upvotes: 0

Related Questions