CODEWITHSUNDEEP
pythonpython-3.xmultiprocessingpython-multiprocessing
Nyxynyx
Nyxynyx

Reputation: 63687

Python Multiprocessing Pool.map Causes Error in __new__

In the simple Python 3 example below where we use the multiproessing module to process the list friends, what is causing the error:

TypeError: new() missing 1 required positional argument: 'name'

No error occurs if simply running 

tom = Friend(tom)
say_hello(tom)

Any ideas how we can solve this issue? Thank you!

Code

import multiprocessing

def say_hello(friend):
    print('Hello', friend.name, '!')

class Friend:
    friends = {}
    def __new__(cls, name):
        if name not in cls.friends:
            cls.friends[name] = super(Friend, cls).__new__(cls)
        return cls.friends[name]

    def __init__(self, name):
        self.name = name

jack = Friend('jack')
ryan = Friend('ryan')
friends = [jack, ryan]
multiprocessing.Pool(2).map(say_hello, friends)

Full Error Trace

Traceback (most recent call last):
  File "/Users/nyxynyx/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/Users/nyxynyx/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/nyxynyx/opt/anaconda3/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/Users/nyxynyx/opt/anaconda3/lib/python3.7/multiprocessing/queues.py", line 354, in get
    return _ForkingPickler.loads(res)
TypeError: __new__() missing 1 required positional argument: 'name'

Upvotes: 3

Views: 2316

Answers (2)

Saim Raza
Saim Raza

Reputation: 1460

Define __reduce__ to make objects of Friend class pickleable (serializable) to be sent to other processes.

import multiprocessing

def say_hello(friend):
    print('Hello', friend.name, '!')

class Friend:
    friends = {}
    def __new__(cls, name):
        if name in cls.friends:
            return cls.friends[name]
        else:
            return super(Friend, cls).__new__(cls)

    def __init__(self, name):
        self.name = name

    def __reduce__(self):
        return self.__class__, (self.name,)

jack = Friend('jack')
ryan = Friend('ryan')
friends = [jack, ryan]
multiprocessing.Pool(2).map(say_hello, friends)

Upvotes: 1

Darkonaut
Darkonaut

Reputation: 21694

It's an error during unpickling because the name wasn't prepared for being passed when recreating the object during unpickling.

It can be already reproduced with:

pickle.loads(pickle.dumps(jack))

Traceback (most recent call last): 
  ...
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-9-239857af5731>", line 1, in <module>
    pickle.loads(pickle.dumps(jack))
TypeError: __new__() missing 1 required positional argument: 'name'

The solution is to implement object.__getnewargs__() or object.__getnewargs_ex__().

object.getnewargs()

This method serves a similar purpose as getnewargs_ex(), but supports only positional arguments. It must return a tuple of arguments args which will be passed to the new() method upon unpickling.

getnewargs() will not be called if getnewargs_ex() is defined.

Changed in version 3.6: Before Python 3.6, getnewargs() was called instead of getnewargs_ex() in protocols 2 and 3.

So in your case:

def __getnewargs__(self):
    return self.name,

Upvotes: 2

Related Questions