CODEWITHSUNDEEP
pythonmultiprocessing
Zanam
Zanam

Reputation: 4807

Understanding python multiprocessing: does code wait for all process to finish

I have a piece of code which looks as following:

Data = [1, 2, 3, 4, 5]
def Fn2(obj):
  return 2*obj

p = Pool(multiprocessing.cpu_count())
valueList = p.map(Fn2, Data) #---question Line 1
valueList = 2*valueList  #--question Line 2

At Line 1 the processors are returning values which gets stored in valueList. I have 8 cores on my computer. Initially job gets assigned to all cores. When let's say out of 8 cores, 5 cores free up. Do the rest of cores jump to Line 2 or wait for all cores to finish the job and fill up the valueList and then proceed to Line 2.

How do I test your answer?

Upvotes: 0

Views: 415

Answers (1)

Alex Hall
Alex Hall

Reputation: 36013

Do the rest of cores jump to Line 2

No.

How do I test your answer?

When in doubt about code running in parallel, add sleeps. When in doubt about code in general, add prints.

import multiprocessing
from time import sleep

from multiprocess import Pool

Data = [1, 2, 3, 4, 5]

def Fn2(obj):
    print(obj)
    sleep(2)
    return 2 * obj

p = Pool(multiprocessing.cpu_count())
valueList = p.map(Fn2, Data)  # ---question Line 1
print(valueList)
valueList = 2 * valueList  # --question Line 2

This gives the following output:

1
2
3
4
5
[2, 4, 6, 8, 10]

And based on the timing it should be clear what's going on. Try tweaking the pool size to 3 or something to experiment further.

Upvotes: 1

Related Questions