CODEWITHSUNDEEP
pythonmultiprocessingthreadpool
Noah
Noah

Reputation: 174

multiprocessing more than 2 arguments to pool.map using partial from itertools

I need to pass 5 arguments to multiprocessing.pool.map. I can succesfully pass 2 using itertools.partial but I need more that way i can use a counter, using a counter requires a lock and shared counter. I looked at one other question on stack, but it didn't apply to me current situation.

Here is my current code using partial:

import multiprocessing
from multiprocessing import Pool, Value, Lock
from itertools import partial

def check(argumentnumberone, iteratorr):
    #MY CODE HERE


def main():
    pool = Pool(numbofthreadshere)
    pool.daemon = True
    func2 = partial(check, goodones)#this is argumentone
    result = pool.map(func2, arrange)#this is the iterator

I need to pass a lock, counter, and at least 1 more argument to check(), how would i go about this, my current code works fine with no errors.

Upvotes: 0

Views: 577

Answers (1)

amirouche
amirouche

Reputation: 7873

Want you can do is prepare the list of arguments and pass a tuple or list of arguments to pool map instead of using partial:


def check(*args):
    print('check', args)


def main():
    pool = Pool(8)
    pool.daemon = True
    result = pool.map(check, [(1, 2, 3), (4, 5, 6)])

main()

Or you can use partial like so:

import multiprocessing

from multiprocessing import Pool, Value, Lock
from functools import partial


def check(*args):
    print('check', args)


def main():
    pool = Pool(8)
    pool.daemon = True
    my_func = partial(check, 'arg0 from partial', 'arg1 from partial')
    result = pool.map(my_func, range(10))

main()

The last snippet will output the following:

check ('arg0 from partial', 'arg1 from partial', 0)
check ('arg0 from partial', 'arg1 from partial', 1)
check ('arg0 from partial', 'arg1 from partial', 2)
check ('arg0 from partial', 'arg1 from partial', 3)
check ('arg0 from partial', 'arg1 from partial', 4)
check ('arg0 from partial', 'arg1 from partial', 5)
check ('arg0 from partial', 'arg1 from partial', 8)
check ('arg0 from partial', 'arg1 from partial', 6)
check ('arg0 from partial', 'arg1 from partial', 9)
check ('arg0 from partial', 'arg1 from partial', 7)

Upvotes: 1

Related Questions