CODEWITHSUNDEEP
python-3.xmultiprocessingpython-multiprocessing
kar
kar

Reputation: 308

Ordering in process pool python

I am trying to return the result from processes running in parallel in python. Here is the code .

from dateutil.parser import parse
from multiprocessing.dummy import Pool
from itertools import *
import time
import random

def abcd(a):
    time.sleep(a)
    print(a)
    return a

p = Pool(3)
a = p.starmap(abcd, zip([10, 2, 15]))
p.close()
p.join()
print(a)

Is the result always returned in the order of the array inside zip ?

Upvotes: 7

Views: 4078

Answers (2)

anthony sottile
anthony sottile

Reputation: 70195

Pool.starmap (and Pool.map) will return the values in order of the arguments passed into them -- this does not however mean that they will be run in order. If you don't care about order and are ok with unordered results (and potentially more performance) there is Pool.imap_unordered.

See the documentation

Upvotes: 15

ahota
ahota

Reputation: 459

There is no guarantee that they will be in the same order because ultimately the OS handles scheduling the processes. They may end up in the same order, and you might be able to increase the chances of it if you set chunksize to 1 in starmap(). Otherwise, at best the results will be in execution order grouped by chunk size.

If you need to know the original order afterwards, the trick I use is to just pass in an index (or a numpy.ndindex for n-dimensional data) that is just re-emitted by the worker function.

Upvotes: 2

Related Questions