CODEWITHSUNDEEP
pythondictionaryparametersmultiprocessing
Camon
Camon

Reputation: 1043

Passing python dictionary of parameters though multiprocessing

I have a dictionary that has dictionaries of parameters stored in them. What I want to do is use multiprocessing to pass the dictionaries of parameters into a function. This is what I would like it to look like:

from multiprocessing import Pool

# Dictionary with dictionaries of parameters:
my_dict = {"one":{"parm1":1, "parm2":2, "parm3":3}, "two":{"parm1":4, "parm2":5, "parm3":6}, "three":{"parm1":7, "parm2":8, "parm3":9}}

def function(parm1, parm2, parm3):  # Function that is called during multiprocessing.
    print(parm1, parm2, parm3)
    print(parm1 * parm2 * parm3)

def store_info(a):  # The callback function.
    print(a)

p = Pool(4)  # Setting up the number of CPU's to use.

for key, value in my_dict.items():  # For loop used to separate keys from values.
    print(value)
    data = p.map_async(function, **value, callback=store_info)

I used **value to indicate that I want those values in the dictionary to be passed though the function. I know it can't be used there in particular.

My question: How can I pass a dictionary of parameters though multiprocessing?

I found some info [over here](Python multiprocessing for each key in dictionary"Python multiprocessing for each key in dictionary"), but I am unable to implement it into my code.

I am using Python 3.4 on Mac OS Yosemite.

Upvotes: 0

Views: 2265

Answers (1)

Tom Dalton
Tom Dalton

Reputation: 6190

You're misusing the map_async - you want to use apply_async instead:

from multiprocessing import Pool

def function(parm1, parm2, parm3):  # Function that is called during multiprocessing.
    print parm1, parm2, parm3
    print parm1 * parm2 * parm3
    return parm1 * parm2 * parm3

def store_info(a):  # The callback function.
    print a

if __name__ == "__main__":
    # Dictionary with dictionaries of parameters:
    my_dict = {"one":{"parm1":1, "parm2":2, "parm3":3}, "two":{"parm1":4, "parm2":5, "parm3":6}, "three":{"parm1":7, "parm2":8, "parm3":9}}

    p = Pool(4)  # Setting up the number of CPU's to use.

    for key, params_dict in my_dict.items():  # For loop used to separate keys from values.
        print params_dict
        p.apply_async(function, args=tuple(), kwds=params_dict, callback=store_info)

    p.close()
    p.join()

Upvotes: 1

Related Questions