CODEWITHSUNDEEP
pythonmultithreadingqueue
Nathan Osman
Nathan Osman

Reputation: 73175

How to get this thread waiting on a queue to quit?

I've got two threads in my application. One that puts values in a Queue, and another that pulls them from the Queue and processes them.

I am faced with a dilemma when shutting the application down. The thread that processes items in the Queue is stuck on:

item = request_queue.get() # this call blocks until an item is available

The only thing that will terminate the thread is if another item is added to the Queue - and since the main thread doesn't add anything (because it's shutting down), the application locks.

So... how can I instruct Queue.get() to somehow return even if there is nothing on the Queue?

Upvotes: 12

Views: 7701

Answers (4)

Vermillion Hue
Vermillion Hue

Reputation: 11

You can check queue size and wait a specific time before exiting queue, so that you know more data items are not being added at the time of exiting.

last_obj_time = time.time()
while True:
    if request_queue.qsize() > 0:
        item = request_queue.get()
        last_obj_time = time.time()

    elif time.time() - last_obj_time > 10:
        break

Upvotes: 0

Nathan Osman
Nathan Osman

Reputation: 73175

The answer it turns out is quite simple. Pick a value that would be invalid for the code that processes the Queue (None is ideal for that) and push that into the Queue. Then have the Queue processing thread quit when it gets the value:

while True:

    item = request_queue.get()

    if item is None:
        break

    # process the Queue as per normal...

Upvotes: 21

user1371492
user1371492

Reputation: 11

This problem can still occur if Queue.get() is called in the main thread - so the setDaemon(True) answer is not a universal solution.

For example this script cannot be stopped with Ctrl-C

#!/usr/bin/python
import Queue

theQ = Queue.Queue()
print "one thread getting from Queue"
print theQ.get()

Rather than putting a timeout on Queue.get() and dealing with exceptions, a simple solution is to do a wait loop until something is there. This script can be killed with Ctrl-C

#!/usr/bin/python
import Queue
theQ = Queue.Queue()
print "one thread getting from Queue"
while theQ.empty():
    time.sleep(0.01) # or whatever value is appropriate for your event arrivals
print theQ.get()

Upvotes: 0

senderle
senderle

Reputation: 150977

Since the blocking thread is not the main thread, you could also set .daemon = True. 

import time
import threading
from Queue import Queue

def getter(q):
    while True:
        print 'getting...'
        print q.get(), 'gotten'

def putter(q):
    for i in range(10):
        time.sleep(0.5)
        q.put(i)
        time.sleep(0.5)

q = Queue()
get_thread = threading.Thread(target=getter, args=(q,))
get_thread.daemon = True
get_thread.start()

putter(q)

Upvotes: 1

Related Questions