Reputation: 63687
In the simple Python 3 example below where we use the multiproessing
module to process the list friends
, what is causing the error:
TypeError: new() missing 1 required positional argument: 'name'
No error occurs if simply running
tom = Friend(tom)
say_hello(tom)
Any ideas how we can solve this issue? Thank you!
Code
import multiprocessing
def say_hello(friend):
print('Hello', friend.name, '!')
class Friend:
friends = {}
def __new__(cls, name):
if name not in cls.friends:
cls.friends[name] = super(Friend, cls).__new__(cls)
return cls.friends[name]
def __init__(self, name):
self.name = name
jack = Friend('jack')
ryan = Friend('ryan')
friends = [jack, ryan]
multiprocessing.Pool(2).map(say_hello, friends)
Full Error Trace
Traceback (most recent call last):
File "/Users/nyxynyx/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Users/nyxynyx/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Users/nyxynyx/opt/anaconda3/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/Users/nyxynyx/opt/anaconda3/lib/python3.7/multiprocessing/queues.py", line 354, in get
return _ForkingPickler.loads(res)
TypeError: __new__() missing 1 required positional argument: 'name'
Upvotes: 3
Views: 2316
Reputation: 1460
Define __reduce__
to make objects of Friend
class pickleable (serializable) to be sent to other processes.
import multiprocessing
def say_hello(friend):
print('Hello', friend.name, '!')
class Friend:
friends = {}
def __new__(cls, name):
if name in cls.friends:
return cls.friends[name]
else:
return super(Friend, cls).__new__(cls)
def __init__(self, name):
self.name = name
def __reduce__(self):
return self.__class__, (self.name,)
jack = Friend('jack')
ryan = Friend('ryan')
friends = [jack, ryan]
multiprocessing.Pool(2).map(say_hello, friends)
Upvotes: 1
Reputation: 21694
It's an error during unpickling because the name
wasn't prepared for being passed when recreating the object during unpickling.
It can be already reproduced with:
pickle.loads(pickle.dumps(jack))
Traceback (most recent call last):
...
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-9-239857af5731>", line 1, in <module>
pickle.loads(pickle.dumps(jack))
TypeError: __new__() missing 1 required positional argument: 'name'
The solution is to implement object.__getnewargs__()
or object.__getnewargs_ex__()
.
object.getnewargs()
This method serves a similar purpose as getnewargs_ex(), but supports only positional arguments. It must return a tuple of arguments args which will be passed to the new() method upon unpickling.
getnewargs() will not be called if getnewargs_ex() is defined.
Changed in version 3.6: Before Python 3.6, getnewargs() was called instead of getnewargs_ex() in protocols 2 and 3.
So in your case:
def __getnewargs__(self):
return self.name,
Upvotes: 2