CODEWITHSUNDEEP
pythonpython-multiprocessing
Daniel Kats
Daniel Kats

Reputation: 5564

Detect when multiprocessing queue is empty and closed

Let's say I have two processes: a reader and a writer. How does the writer detect when the reader has finished writing values?

The multiprocessing module has a queue with a close method that seems custom-built for this purpose. But how do you detect when the queue has been closed?

This doesn't seem to work, as the writer never exits:

import multiprocessing as mp

def getter(q):
    while True:
        try:
            print(q.get())
        except Exception:
            break

def putter(q):
    q.put(1)
    q.put(2)
    q.close()

q = mp.Queue()
writer = mp.Process(target=putter, args=(q, ))
reader = mp.Process(target=getter, args=(q, ))
reader.start()
writer.start()

writer.join()
reader.join()

Should the reader-writer pair use a sentinel value to signal end of writing? Then what's the point of having the close method?

EDIT: While this question asks about the Queue module (now queue), I am asking specifically about mp.Queue and what the correct use of the .close method is.

Upvotes: 7

Views: 6816

Answers (1)

user2357112
user2357112

Reputation: 282006

But how do you detect when the queue has been closed?

You don't. That is not the purpose of close. Calling close doesn't even guarantee that no more items will be added to the queue; the docs say

Indicate that no more data will be put on this queue by the current process.

close is intended to shut down the current process's feeder thread for that queue (or at least start shutting it down), not to communicate an end-of-queue to other processes.

If you want to signal that no more values will be written to the queue, use standard techniques like enqueueing a sentinel object, like you would with an ordinary queue.Queue.

Upvotes: 13

Related Questions