CODEWITHSUNDEEP
pythonpython-2.7multiprocessing
Curious2learn
Curious2learn

Reputation: 33598

Obtain results from processes using python multiprocessing

I am trying to understand how to use the multiprocessing module in Python. The code below spawns four processes and outputs the results as they become available. It seems to me that there must be a better way for how the results are obtained from the Queue; some method that does not rely on counting how many items the Queue contains but that just returns items as they become available and then gracefully exits once the queue is empty. The docs say that Queue.empty() method is not reliable. Is there a better alternative for how to consume the results from the queue?

import multiprocessing as mp
import time


def multby4_wq(x, queue):
    print "Starting!"
    time.sleep(5.0/x)
    a = x*4
    queue.put(a)


if __name__ == '__main__':
    queue1 = mp.Queue()
    for i in range(1, 5):
        p = mp.Process(target=multbyc_wq, args=(i, queue1))
        p.start()
    for i in range(1, 5): # This is what I am referring to as counting again
        print queue1.get()

Upvotes: 0

Views: 210

Answers (1)

falsetru
falsetru

Reputation: 368904

Instead of using queue, how about using Pool?

For example,

import multiprocessing as mp
import time


def multby4_wq(x):
    print "Starting!"
    time.sleep(5.0/x)
    a = x*4
    return a

if __name__ == '__main__':
    pool = mp.Pool(4)
    for result in pool.map(multby4_wq, range(1, 5)):
        print result

Pass multiple arguments

Assume you have a function that accept multiple parameters (add in this example). Make a wrapper function that pass arguments to add (add_wrapper).

import multiprocessing as mp
import time


def add(x, y):
    time.sleep(1)
    return x + y

def add_wrapper(args):
    return add(*args)

if __name__ == '__main__':
    pool = mp.Pool(4)
    for result in pool.map(add_wrapper, [(1,2), (3,4), (5,6), (7,8)]):
        print result

Upvotes: 2

Related Questions