CODEWITHSUNDEEP
pythonpython-3.xmultiprocessingpython-3.6
DachuanZhao
DachuanZhao

Reputation: 1339

A strange result in python using function multiprocessing.Pool?

Here is code:

one:

import multiprocessing
import datetime
import time
a = [i for i in range(100)]
res1 = []
def jia(a,b,c):
    time.sleep(0.1)
    return a + b + c
res1_time_start = datetime.datetime.now()
p = multiprocessing.Pool(4)
res1 = [p.apply_async(jia,(i,1,1)) for i in a]
res1 = [i.get() for i in res1]
res1_time_end = datetime.datetime.now()
print((res1_time_end - res1_time_start).seconds)

the other:

import multiprocessing
import datetime
import time
a = [i for i in range(100)]
res1 = []
def jia(a,b,c):
    time.sleep(0.1)
    return a + b + c
res1_time_start = datetime.datetime.now()
p = multiprocessing.Pool(4)
for i in a:
    res1.append(p.apply_async(jia,(i,1,1)).get())
res1_time_end = datetime.datetime.now()
print((res1_time_end - res1_time_start).seconds)

The strange result is that the former cost 2.5s but the latter cost 10s. Why?Does list.append Blocks process?

Upvotes: 1

Views: 79

Answers (1)

Tim Peters
Tim Peters

Reputation: 70592

No, append() doesn't block, but .get() does. It's the purpose of .get() to block. In

res1 = [p.apply_async(jia,(i,1,1)) for i in a]
res1 = [i.get() for i in res1]

all the work is given to the pool by the first line before any .get() is invoked by the second line. But in

for i in a:
    res1.append(p.apply_async(jia,(i,1,1)).get())

the .get() waits for a single piece of work to finish before it allows the loop to continue to even start the next piece of work. You've serialized the work - nothing useful happens in parallel.

Upvotes: 5

Related Questions