CODEWITHSUNDEEP
pythonmultiprocessing
Kaushik Balamukundhan
Kaushik Balamukundhan

Reputation: 555

Python Multiprocessing queue

I am populating a queue with a set of jobs that I want to run in parallel and using python's multiprocessing module for doing that. Code snippet below:

import multiprocessing
from multiprocessing import Queue
queue = Queue()
jobs = [['a', 'b'], ['c', 'd']]
for job in jobs:
    queue.put(job)

When I do queue.get() I get the following:

['a', 'b']

Why is the queue not getting populated with all the jobs?

Upvotes: 5

Views: 11334

Answers (2)

Mahmoud Abdelkader
Mahmoud Abdelkader

Reputation: 24999

The queue is getting populated with all your jobs. queue.get() will

Remove and return an item from the queue.

An item is singular. If you want to drain the queue, then just put your .get() in a loop, but be sure to catch the Empty exception.

Upvotes: 12

Utku Zihnioglu
Utku Zihnioglu

Reputation: 4883

The queue is actually geting populated. You need to call queue.get() for each time you put an object to the queue. So you just need to call queue.get() one more time.

>>> import multiprocessing
>>> from multiprocessing import Queue
>>> queue = Queue()
>>> jobs = [['a', 'b'], ['c', 'd']]
>>> for job in jobs:
    queue.put(job)


>>> queue.get()
['a', 'b']
>>> queue.get()
['c', 'd']

Upvotes: 15

Related Questions