Reputation: 55
I want to save the task result in to a queue, the tasks is run in multiple process. After all tasks running done, read result from the queue.
Why the queue size printed in child task is right, but in parent process is 0? Also I can not get the queue data in parent. How to resolve this?
Code in Python3:
import multiprocessing
que = multiprocessing.Queue()
def task(n):
que.put(n)
print("Child: queue len is %d" % que.qsize())
if __name__ == '__main__':
pool = multiprocessing.Pool()
pool.map(task, range(10))
print("Parent: queue len is %d" % que.qsize())
Output:
Child: queue len is 1
Child: queue len is 2
Child: queue len is 3
Child: queue len is 4
Child: queue len is 5
Child: queue len is 6
Child: queue len is 7
Child: queue len is 8
Child: queue len is 9
Child: queue len is 10
Parent: queue len is 0 // Why here is not 10??
Upvotes: 1
Views: 596
Reputation: 19332
When you run task
in pool.map
, the module containing function task
is imported and task
is called with arguments you specified in pool.map
.
So, the original queue is not passed to it. When you import the module (in the other process), a new global queue is created.
To make the behaviour more apparent, I modified you example a bit, so that it spawns more processes:
import multiprocessing
import time
que = multiprocessing.Queue()
def task(n):
que.put(n)
print("Child: queue len is %d" % que.qsize())
time.sleep(0.1)
if __name__ == '__main__':
pool = multiprocessing.Pool(10)
pool.map(task, range(10))
print("Parent: queue len is %d" % que.qsize())
Now it prints:
Child: queue len is 1
Child: queue len is 1
Child: queue len is 2
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Parent: queue len is 0
To share the queue, you would have to pass it to the other processes, but cannot be done through map
. There alternatives offered in several other SO answers, e.g.
What is best in your case depends on what you are actually trying to do with that queue.
Upvotes: 1