CODEWITHSUNDEEP
pythonmultithreadingpython-3.xmultiprocessing
shao
shao

Reputation: 11

Python multiprocessing.Pool: how to join the reasults in a parallel way?

I have read Python multiprocessing.Pool: when to use apply, apply_async or map? it was useful, but still had my own questions. In the following code, I want result_list.append(result) in a parallel way, I want the 4 processors to append the result parallelly and convert 4 lists to 1 list.

import multiprocessing as mp
import time

def foo_pool(x):
    time.sleep(2)
    return x*x

result_list = []
def log_result(result):
    # This is called whenever foo_pool(i) returns a result.
    # result_list is modified only by the main process, not the pool workers.
    result_list.append(result)

def apply_async_with_callback():
    pool = mp.Pool(4)
    for i in range(10):
        pool.apply_async(foo_pool, args = (i, ), callback = log_result)
    pool.close()
    pool.join()
    print(result_list)

if __name__ == '__main__':
    apply_async_with_callback()

Upvotes: 1

Views: 4924

Answers (1)

Wenlong Liu
Wenlong Liu

Reputation: 444

Multiprocessing pool will be your choice.

Following are some sample code, hope it helps. You can also check another my answer to see more details. How can I make my python code run faster

    from multiprocessing import Pool
    import time

    def foo_pool(x):
        return x*x

    def main():
         pool = Pool(4)
         sampleData = [x for x in range(9)]
         results = pool.map(foo_pool, sampleData)
         pool.close()
         pool.join()
         print(results)

    if __name__ == '__main__':
         main()

Upvotes: 2

Related Questions