CODEWITHSUNDEEP
python-3.xmultiprocessingapplypython-multiprocessing
Shihab Khan
Shihab Khan

Reputation: 201

Ordered results with apply_async

I have read that the function apply_async doesn't give ordered results. If I have repeated calls to a function which prints the squares of a list of numbers, I can see from the display that the list is not ordered.

However when the function returns the number instead of printing it and I use .get() to get the values, then I see that the results are ordered.

I have a few questions --

  1. Why are the results from .get() ordered?
  2. If I have a loop which as a variable named a and its value is different for different iterations. Will using apply_async cause overwrites of the values of a as it runs the processes in parallel and asynchronously?
  3. Will I be able to save computational time if I run apply instead of apply_async? My code shows that apply is slower than the for loop. Why is that so?
  4. Can we use a function declared within the ___main___ function with apply_async?

Here is a small working example:

from multiprocessing import Pool
import time

def f(x):
    return x*x

if __name__ == '__main__':
    
    print('For loop')
    t1f = time.time()
    for ii in range(20):
        f(ii)
    t2f = time.time()    
    print('Time taken for For loop = ', t2f-t1f,' seconds')
    
    pool = Pool(processes=4)
    print('Apply async loop')
    t1a = time.time()
    results = [pool.apply_async(f, args = (j,)) for j in range(20)]

    pool.close()
    pool.join()
    t2a = time.time()    
    print('Time taken for pool = ', t2a-t1a,' seconds')
    print([results[hh].get() for hh in range(len(results))])

This results as:

For loop Time taken for For loop = 5.9604644775390625e-06 seconds

Apply async loop Time taken for pool = 0.10188460350036621 seconds

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361]

Upvotes: 0

Views: 431

Answers (1)

georgexsh
georgexsh

Reputation: 16634

  1. Why are the results from .get() ordered?

because the results list is ordered.

  1. If I have a loop which as a variable named a and its value is different for different iterations. Will using apply_async cause overwrites of the values of a as it runs the processes in parallel and asynchronously?

generally no, but I can't tell without seeing the code.

  1. Will I be able to save computational time if I run apply instead of apply_async? My code shows that apply is slower than the for loop. Why is that so?

no, apply blocks on each call, there is no parallelism. apply is slower because of multiprocessing overhead.

  1. Can we use a function declared within the ___main___ function with apply_async?

yes for *nix, no for windows, because there is no fork().

your time measurement of .apply_async is wrong, you should take t2a after result.get, and don't assume the result is finished in order:

while not all(r.ready() for r in results):
    time.sleep(0.1)

btw, your work function runs too fast to finish, do more computation to do a true benchmark.

Upvotes: 4

Related Questions