CODEWITHSUNDEEP
python-3.xpython-multithreading
chirinosky
chirinosky

Reputation: 4538

Hanging Threads and Exceptions

The code below is supposed to authenticate to an FTP server and check if write access is allowed. It runs fine whenever a single host is used. 

import ftplib
import queue
import threading


def check_ftp(options):
    host = options.get('host')
    user = options.get('user')
    passwd = options.get('passwd')
    port = options.get('port')
    # Attempt to connect and authenticate
    ftp = ftplib.FTP()
    try:
        ftp.connect(host, port, timeout=5)
        ftp.login(user, passwd)
        print('{}:{} - Login successful.'.format(host, port))
    except(ftplib.error_perm, OSError) as e:
        print('{}:{} - Unable to connect/auth: {}'.format(host, port, e))
        return
    # Attempt to write & get dir listing
    is_writable = False
    contents = []
    try:
        test_folder = 'test_folder'
        ftp.mkd(test_folder)
        print('{}:{} - FTP root is writable'.format(host, port))
        is_writable = True
        ftp.retrlines('LIST', contents.append)
        ftp.rmd(test_folder)
    except ftplib.error_perm as e:
        ftp.retrlines('LIST', contents.append)
        print('{}:{} - Not writable: {}'.format(host, port, e))
    ftp.quit()


class ModuleRunner(threading.Thread):

    def __init__(self, queue):
        threading.Thread.__init__(self)
        self.queue = queue

    def run(self):
        while True:
            # gets the options from the queue
            options = self.queue.get()
            # run the module
            check_ftp(options)

            # Let queue know the job is one
            self.queue.task_done()


def run(thread_count):
    options =[ 
        {'host': 'ftp.uconn.edu',
         'user': 'anonymous',
         'passwd': '[email protected]',
         'port': 21},
         {'host': 'speedtest.tele2.net',
         'user': 'anonymous',
         'passwd': '[email protected]',
         'port': 21},
         {'host': 'test.talia.net',
         'user': 'anonymous',
         'passwd': '[email protected]',
         'port': 21},
    ]
    # Fill queue
    q = queue.Queue()
    for opt in options:
        q.put(opt)
    # Create a thread pool
    threads = thread_count
    for i in range(threads):
        t = ModuleRunner(q)
        t.setDaemon(True)
        t.start()
    q.join()

if __name__ == '__main__':
    run(1)

However, if 03 or more hosts are added (see options variable), the script crashes and hangs every time, even when one thread is used. Errors:

ftp.uconn.edu:21 - Login successful.
ftp.uconn.edu:21 - Not writable: 550 test_folder: Permission denied
speedtest.tele2.net:21 - Login successful.
speedtest.tele2.net:21 - Not writable: 550 Permission denied.
test.talia.net:21 - Login successful.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "ftp.py", line 25, in check_ftp
    ftp.mkd(test_folder)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ftplib.py", line 641, in mkd
    resp = self.voidcmd('MKD ' + dirname)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ftplib.py", line 276, in voidcmd
    return self.voidresp()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ftplib.py", line 249, in voidresp
    resp = self.getresp()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ftplib.py", line 244, in getresp
    raise error_perm(resp)
ftplib.error_perm: 550 test_folder: Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "ftp.py", line 47, in run
    check_ftp(options)
  File "ftp.py", line 31, in check_ftp
    ftp.retrlines('LIST', contents.append)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ftplib.py", line 466, in retrlines
    with self.transfercmd(cmd) as conn, \
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ftplib.py", line 397, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ftplib.py", line 359, in ntransfercmd
    source_address=self.source_address)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 722, in create_connection
    raise err
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)

I don't understand why the socket times out, even when the checks run under one thread. Also, how can a thread be terminated if it hangs for too long. Thank you in advance.

Upvotes: 0

Views: 194

Answers (1)

TinyTheBrontosaurus
TinyTheBrontosaurus

Reputation: 4820

First thing is does this work:

def run():
    check_ftp(options[0])
    check_ftp(options[1])
    check_ftp(options[2])

I'm guessing no. It appears that the FTP object is not being cleaned up correctly. Using the with operation should make it easier to close FTP. See this example

from ftplib import FTP
with FTP("ftp1.at.proftpd.org") as ftp:
    ftp.login()
    ftp.dir()

Also, threads cannot be cancelled, but processes can. Switching to the multiprocessing library will allow them to be killed.

Upvotes: 1

Related Questions