CODEWITHSUNDEEP
python-3.xmultiprocessingpathos
Indominus
Indominus

Reputation: 1258

Does pathos.multiprocessing have starmap?

I got an error when executing code below. The problem seems to be map doesn't support functions taking multiple inputs, just as in the python built-in multiprocessing package. But in the built-in package, there is a starmap that solves this issue. Does pathos.multiprocessing have the same?

import pathos.multiprocessing as mp


class Bar:
    def foo(self, name):
        return len(str(name))

    def boo(self, x, y, z):
        sum = self.foo(x)
        sum += self.foo(y)
        sum += self.foo(z)
        return sum


if __name__ == '__main__':
    b = Bar()
    pool = mp.ProcessingPool()
    results = pool.map(b.boo, [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
    print(results)

TypeError: boo() missing 2 required positional arguments: 'y' and 'z'

Update for lambda expression as suggested (didn't work):

if __name__ == '__main__':
    b = Bar()
    pool = mp.ProcessingPool()
    results = pool.map(lambda x: b.boo(*x), [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
    print(results)

multiprocess.pool.RemoteTraceback:

"""

Traceback (most recent call last):

File "C:\Users\yg451\Anaconda3\lib\site-packages\multiprocess\pool.py", line 121, in worker

result = (True, func(*args, **kwds))

File "C:\Users\yg451\Anaconda3\lib\site-packages\multiprocess\pool.py", line 44, in mapstar

return list(map(*args))

File "C:\Users\yg451\Anaconda3\lib\site-packages\pathos\helpers\mp_helper.py", line 15, in

func = lambda args: f(*args)

File "C:/Users/yg451/Code/foo/Machine Learning/xPype/test/scratch.py", line 18, in

results = pool.map(lambda x: b.boo(*x), [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])

NameError: name 'b' is not defined

"""

Upvotes: 5

Views: 3758

Answers (1)

Mike McKerns
Mike McKerns

Reputation: 35247

I'm the pathos author. pathos is older than starmap, and doesn't really need it. It solved multiple arguments in a pool exactly the same way that the built-in map does.

>>> import pathos.multiprocessing as mp
>>> class Bar:
...     def foo(self, name):
...         return len(str(name))
...     def boo(self, x, y, z):
...         sum = self.foo(x)
...         sum += self.foo(y)
...         sum += self.foo(z)
...         return sum
... 
>>> b = Bar()
>>> pool = mp.ProcessingPool()
>>> f = lambda x: b.boo(*x)    
>>> results = pool.map(f, [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
>>> results 
[6, 4, 5]
>>> results = pool.map(b.boo, [12, 9, 'a'], [3, 9, 'b'], [456, 10, 'cde'])
>>> results
[6, 4, 5]
>>> results = map(b.boo, [12, 9, 'a'], [3, 9, 'b'], [456, 10, 'cde'])
>>> list(results)
[6, 4, 5]
>>>

So, essentially, starmap is unnecessary. However, as it's been recently added to the standard Pool interface in multiprocessing in certain versions of python, it probably should be more prominent in pathos. Note that it already is possible to get to an "augmented" version of starmap from pathos if you like.

>>> import pathos
>>> mp = pathos.helpers.mp
>>> p = mp.Pool()
>>> p.starmap
<bound method Pool.starmap of <multiprocess.pool.Pool object at 0x1038684e0>>
>>>

Upvotes: 10

Related Questions