CODEWITHSUNDEEP
pythonmultithreading
j0ker
j0ker

Reputation: 4139

Deadlock in Python Threads

I am trying to implement a simpley portscanner with Python. It works by creating a number of worker threads which scan ports that are provided in a queue. They save the results in another queue. When all ports are scanned the threads and the application should terminate. And here lies the problem: For small numbers of ports everything works fine, but if I try to scan 200 or more ports, the application will get caught in a deadlock. I have no idea, why.

class ConnectScan(threading.Thread):
    def __init__(self, to_scan, scanned):
        threading.Thread.__init__(self)
        self.to_scan = to_scan
        self.scanned = scanned

    def run(self):
        while True:
            try:
                host, port = self.to_scan.get()
            except Queue.Empty:
                break
            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            try:
                s.connect((host, port))
                s.close()
                self.scanned.put((host, port, 'open'))
            except socket.error:
                self.scanned.put((host, port, 'closed'))
            self.to_scan.task_done()


class ConnectScanner(object):
    def scan(self, host, port_from, port_to):
        to_scan = Queue.Queue()
        scanned = Queue.Queue()
        for port in range(port_from, port_to + 1):
            to_scan.put((host, port))
        for i in range(20):
            ConnectScan(to_scan, scanned).start()
        to_scan.join()

Does anybody see what might be wrong? Also I would appreciate some tipps how to debug such threading issues in Python.

Upvotes: 2

Views: 2375

Answers (2)

babbageclunk
babbageclunk

Reputation: 8741

I don't see anything obviously wrong with your code, but as it stands the break will never be hit - self.to_scan.get() will wait forever rather than raising Queue.Empty. Given that you're loading up the queue with ports to scan before starting the threads, you can change that to self.to_scan.get(False) to have the worker threads exit correctly when all the ports have been claimed.

Combined with the fact that you have non-daemon threads (threads that will keep the process alive after the main thread finishes), that could be the cause of the hang. Try printing something after the to_scan.join() to see whether it's stopped there, or at process exit.

As Ray says, if an exception other than socket.error is raised between self.to_scan.get() and self.to_scan.task_done(), then the join call will hang. It could help to change that code to use a try/finally to be sure:

def run(self):
    while True:
        try:
            host, port = self.to_scan.get(False)
        except Queue.Empty:
            break

        try:
            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            try:
                s.connect((host, port))
                s.close()
                self.scanned.put((host, port, 'open'))
            except socket.error:
                self.scanned.put((host, port, 'closed'))
        finally:
            self.to_scan.task_done()

In general, debugging multithreaded processes is tricky. I try to avoid anything blocking indefinitely - it's better to have something crash noisily because a timeout was too short than to have it just stop forever waiting for an item that will never appear. So I'd specify timeouts for your self.to_scan.get, socket.connect and to_scan.join calls.

Use logging to work out the order events are occurring - printing can get interleaved from different threads, but loggers are thread-safe.

Also, something like this recipe can be handy for dumping the current stack trace for each thread.

I haven't used any debuggers with support for debugging multiple threads in Python, but there are some listed here.

Upvotes: 3

Ray
Ray

Reputation: 1869

It is likely that not all items on the to_scan queue are consumed and that you're not calling the task_done method enough times to unblock ConnectScanner.

Could it be that an exception is thrown during the runtime of ConnectScan.run that you're not catching and your threads prematurely terminate?

Upvotes: 1

Related Questions