CODEWITHSUNDEEP
pythonmultithreadingqueue
Andrey Reeshkov
Andrey Reeshkov

Reputation: 361

How to handle exception in threading with queue in Python?

This is never print: "Exception in threadfuncqueue handled by threadfuncqueue", "Exception in threadfuncqueue handled by main thread" and "thread test with queue passed". Never quitting!

from threading import Thread
from Queue import Queue
import time

class ImRaiseError():
    def __init__(self):
        time.sleep(1)
        raise Exception(self.__class__.__name__)

# place for paste worked code example from below 

print "begin thread test with queue"
def threadfuncqueue(q):
    print "\n"+str(q.get())
    while not q.empty():
        try:
            testthread = ImRaiseError()
        finally:
            print "Exception in threadfuncqueue handled by threadfuncqueue"

q = Queue()
items = [1,2]
for i in range(len(items)):
     t = Thread(target=threadfuncqueue,args=(q,))
     if(1 == i):
        t.daemon = False
     else:
        t.daemon = True
     t.start()
for item in items:
    q.put("threadfuncqueue"+str(item))

try:
    q.join()       # block until all tasks are done
finally:
    print "Exception in threadfuncqueue handled by main thread"
print "thread test with queue passed"
quit()

How handle this exception?

Example of worked code, but without queue:

print "=========== procedure style test"
def threadfunc(q):
    print "\n"+str(q)
    while True:
        try:
            testthread = ImRaiseError()
        finally:
            print str(q)+" handled by process"

try:
    threadfunc('testproc')
except Exception as e:
    print "error!",e
print "procedure style test ==========="


print "=========== simple thread tests"
testthread = Thread(target=threadfunc,args=('testthread',))
testthread.start()
try:
    testthread.join()
finally:
    print "Exception in testthread handled by main thread"
testthread1 = Thread(target=threadfunc,args=('testthread1',))
testthread1.start()
try:
    testthread1.join()
finally:
    print "Exception in testthread1 handled by main thread"
print "simple thread tests ==========="

Upvotes: 1

Views: 2101

Answers (1)

skrrgwasme
skrrgwasme

Reputation: 9633

Short Answer

You're putting things in a queue and retrieving them, but if you're going to join a queue, you need to mark tasks as done as you pull them out of the queue and process them. According to the docs, every time you enqueue an item, a counter is incremented, and you need to call q.task_done() to decrement that counter. q.join() will block until that counter reaches zero. Add this immediately after your q.get() call to prevent main from being blocked:

q.task_done()

Also, I find it odd that you're checking q for emptiness after you've retrieved something from it. I'm not sure exactly what you're trying to achieve with that so I don't have any recommendations for you, but I would suggest reconsidering your design in that area.

Other Thoughts

Once you get this code working you should take it over to Code Review because it is a bit of a mess. Here are a few thoughts for you:

Exception Handling

You're not actually "handling" the exception in threadfuncqueue(q). All the finally statement does is allow you to execute cleanup code in the event of an exception. It does not actually catch and handle the exception. The exception will still travel up the call stack. Consider this example, test.py:

try:
    raise Exception
finally:
    print("Yup!")
print("Nope!")

Output:

Yup!
Traceback (most recent call last):
    File "test.py", line 2, in
        raise Exception
Exception

Notice that "Yup!" got printed while "Nope!" didn't. The code in the finally block was executed, but that didn't stop the exception from propagating up the stack and halting the interpreter. You need the except statement for that:

try:
    raise Exception
except Exception: # only catch the exceptions you expect
    print("Yup!")
print("Nope!")

Output:

Yup!
Nope!

This time both are printed, because we caught and handled the exception.

Exception Raising

Your current method of raising the exception in your thread is needlessly complicated. Instead of creating the whole ImRaiseError class, just raise the exception you want with a string:

raise Exception('Whatever error message I want')

If you find yourself manually manipulating mangled names (like self.__class__.__name__), you're usually doing something wrong.

Extra Parentheses

Using parentheses around conditional expressions is generally frowned upon in Python:

if(1 == i): # unnecessary extra characters

Try to break the C/C++/Java habit and get rid of them:

if 1 == i:

Other

I've already gone beyond the scope of this question, so I'm going to cut this off now, but there are a few other things you could clean up and make more idiomatic. Head over to Code Review when you're done here and see what else can be improved.

Upvotes: 2

Related Questions