CODEWITHSUNDEEP
pythonpython-multiprocessing
jcxl
jcxl

Reputation: 155

Error while loading multiprocessing.manager.dictionary from pickle file

I have got an error while loading a multiprocessing.manager.dictionary from pickle. I am sure pickle file exists in the directory where I run python.

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import multiprocessing
>>> a = open("test.pkl", "rb")
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 879, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 733, in __init__
    self._incref()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 783, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 304, in SocketClient
    s.connect(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1165, in load_put
    self.memo[self.readline()[:-1]] = self.stack[-1]
IndexError: list index out of range

Here is the part of the code for saving it into pickles.

from multiprocessing import Manager, Pool
import pickle
from functools import partial

def output(dic, s):
    a = open(s + ".pkl", "wb")
    pickle.dump(dic, a)
    a.close()

data_list = [1,2,3,4]
pool = Pool(processes = 4)
m = Manager()
lock = m.Lock()
dic1 = m.dict()
func = partial(f, dic1) # f is a function that takes 3 arguments with the returned result stored in dic1
pool.map(func, data_list)
output(dic1, "test")
pool.close()
pool.join()

If I print the dictionary out before the output, the result on the terminal looks fine.

Upvotes: 5

Views: 3336

Answers (1)

mata
mata

Reputation: 69042

Manager.dict() doesn't actually return a dict but a proxy object to the real dictionary managed by the manager in a different process. When you call a method on that object, the call is forwarded to the manager where the actual dictionary resides.

As communication between the manager and the client process happens using the pickle protocol, these objects are pickleable and when unpickled produce a proxy to the actual object living inside the manager process again, that's why you see an attempt to create a connection when you try to unpickle the dict.

So if you want to pickle the content of a manager dict, convert it to an actual dict before:

...
pickle.dump(dict(dic), a)
...

Upvotes: 14

Related Questions