CODEWITHSUNDEEP
pythonmultithreadingqueue
Mocking
Mocking

Reputation: 1894

Queue with Thread hangs when worker crashes

I am using Python Queue with Thread. I noticed when a worker crashes, the script hangs and doesn't let me terminate. The following is an example:

from Queue import Queue
from threading import Thread

num_worker_threads = 2

def worker():
        while True:
                item = q.get() 
                1/item
                q.task_done()
q = Queue()
for i in range(num_worker_threads):
        t = Thread(target=worker)
        t.daemon = True
        t.start()

q.put(0)
q.join()

Upvotes: 0

Views: 1378

Answers (3)

Mocking
Mocking

Reputation: 1894

I fixed this by wrapping the job in an exception. I would have thought that when a worker crashes, the script would exit. This is not the case. It looks like q.task_done() never gets called so it hangs on q.join().

Solution:

from Queue import Queue
from threading import Thread

num_worker_threads = 2

def worker():
        while True:
                item = q.get()
                try:
                        1/item
                except Exception as e:
                        print e
                finally:
                        q.task_done()
q = Queue()
for i in range(num_worker_threads):
        t = Thread(target=worker)
        t.daemon = True
        t.start()

q.put(0)
q.join()

Added suggestion by rsy.

Upvotes: 1

rsy
rsy

Reputation: 192

An even cleaner approach would be to have task_done() in the finally block instead.

from Queue import Queue
from threading import Thread

num_worker_threads = 2

def worker():
        while True:
                item = q.get()
                try:
                        1/item
                except Exception as e:
                        print e
                finally: 
                      q.task_done()
q = Queue()
for i in range(num_worker_threads):
        t = Thread(target=worker)
        t.daemon = True
        t.start()

q.put(0)
q.join()

Upvotes: 1

Pythonista
Pythonista

Reputation: 11615

You're calling while True which is continuously trying to get elements from the Queue by using q.get() in your thread, but there may not be any items in the queue. Which will throw an Empty exception since there's nothing in the queue to actually get.

Your loop should be while not q.empty(): or you should be catching the Queue.Empty exception.

Upvotes: 0

Related Questions