Python FTP implicit TLS connection issue

Question

I have a need to connect to FTPS server to which I am able to connect successfully using lftp. However, when I try with Python ftplib.FTP_TLS, it times out, the stack trace shows that it is waiting for the server to send welcome message or like. Does anyone know what the issue is and how to overcome? I wonder if there is something needs to be done on server side, but how come lftp client is working fine. Any help is greatly appreciated.

Here is the stack trace:

    ftp = ftplib.FTP_TLS()  
    ftp.connect(cfg.HOST, cfg.PORT, timeout=60)
  File "C:\Users\username\Softwares\Python27\lib\ftplib.py", line 135, in connect  
    self.welcome = self.getresp()  
  File "C:\Users\username\Softwares\Python27\lib\ftplib.py", line 210, in getresp  
    resp = self.getmultiline()  
  File "C:\Users\username\Softwares\Python27\lib\ftplib.py", line 196, in getmultiline  
    line = self.getline()  
  File "C:\Users\username\Softwares\Python27\lib\ftplib.py", line 183, in getline  
    line = self.file.readline()  
  File "C:\Users\username\Softwares\Python27\lib\socket.py", line 447, in readline  
    data = self._sock.recv(self._rbufsize)  
socket.timeout: timed out  

A successful login using lftp to the same ftps server:

$ lftp
lftp :~> open ftps://ip_address:990
lftp ip_address:~> set ftps:initial-prot P
lftp ip_address:~> login ftps_user_id  ftps_user_passwd
lftp sftp_user_id@ip_address:~> ls
ls: Fatal error: SSL_connect: self signed certificate
lftp ftps_user_id@ip_address:~> set ssl:verif-certificate off
lftp ftps_user_id@ip_address:~> ls
lftp ftps_user_id@ip_address:/>

BTW, I am using Python 2.7.3. I did quite a bit of search using Google but have not found anything helpful.

I am still having this issue, appreciate if someone can help. On looking closely the FTP.connect() the connection to server is not a problem but getting acknowledgement (or the welcome message) from server is an issue. lftp does not have this issue and FileZilla does not have any issue either as in the log here -

Status: Connecting to xx.xx.xx.xxx:990...  
Status: Connection established, initializing TLS...  
Status: Verifying certificate...  
Status: TLS/SSL connection established, waiting for welcome message...  
Response:   220-      Vous allez vous connecter sur un serveur prive  
Response:   220-     Seules les personnes habilitees y sont autorisees  
Response:   220 Les contrevenants s'exposent aux poursuites prevues par la loi.  
Command:    USER xxxxxxxxxxxxx  
Response:   331 Password required for xxxxxxxxxxxxx.  
Command:    PASS **********  
Response:   230 Login OK. Proceed.  
Command:    PBSZ 0  
Response:   200 PBSZ Command OK. Protection buffer size set to 0.  
Command:    PROT P  
Response:   200 PROT Command OK. Using Private data connection  
Status: Connected  
Status: Retrieving directory listing...  
Command:    PWD  
Response:   257 "/" is current folder.  
Command:    TYPE I  
Response:   200 Type set to I.  
Command:    PASV  
Response:   227 Entering Passive Mode (81,93,20,199,4,206).  
Command:    MLSD  
Response:   150 Opening BINARY mode data connection for MLSD /.  
Response:   226 Transfer complete. 0 bytes transferred. 0 bps.  
Status: Directory listing successful  

Answer 1

Here's an implementation a little bit more "industrial".

Let's notice that in the previous examples the property named 'context' was missing in the init.

The code below works perfectly both with Python 2.7 and Python 3.

The code

import ftplib, socket, ssl
FTPTLS_OBJ = ftplib.FTP_TLS

# Class to manage implicit FTP over TLS connections, with passive transfer mode
# - Important note:
#   If you connect to a VSFTPD server, check that the vsftpd.conf file contains
#   the property require_ssl_reuse=NO
class FTPTLS(FTPTLS_OBJ):

    host = "127.0.0.1"
    port = 990
    user = "anonymous"
    timeout = 60

    logLevel = 0

    # Init both this and super
    def __init__(self, host=None, user=None, passwd=None, acct=None, keyfile=None, certfile=None, context=None, timeout=60):        
        FTPTLS_OBJ.__init__(self, host, user, passwd, acct, keyfile, certfile, context, timeout)

    # Custom function: Open a new FTPS session (both connection & login)
    def openSession(self, host="127.0.0.1", port=990, user="anonymous", password=None, timeout=60):
        self.user = user
        # connect()
        ret = self.connect(host, port, timeout)
        # prot_p(): Set up secure data connection.
        try:
            ret = self.prot_p()
            if (self.logLevel > 1): self._log("INFO - FTPS prot_p() done: " + ret)
        except Exception as e:
            if (self.logLevel > 0): self._log("ERROR - FTPS prot_p() failed - " + str(e))
            raise e
        # login()
        try:
            ret = self.login(user=user, passwd=password)
            if (self.logLevel > 1): self._log("INFO - FTPS login() done: " + ret)
        except Exception as e:
            if (self.logLevel > 0): self._log("ERROR - FTPS login() failed - " + str(e))
            raise e
        if (self.logLevel > 1): self._log("INFO - FTPS session successfully opened")

    # Override function
    def connect(self, host="127.0.0.1", port=990, timeout=60):
        self.host = host
        self.port = port
        self.timeout = timeout
        try:
            self.sock = socket.create_connection((self.host, self.port), self.timeout)
            self.af = self.sock.family
            self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)
            self.file = self.sock.makefile('r')
            self.welcome = self.getresp()
            if (self.logLevel > 1): self._log("INFO - FTPS connect() done: " + self.welcome)
        except Exception as e:
            if (self.logLevel > 0): self._log("ERROR - FTPS connect() failed - " + str(e))
            raise e
        return self.welcome

    # Override function
    def makepasv(self):
        host, port = FTPTLS_OBJ.makepasv(self)
        # Change the host back to the original IP that was used for the connection
        host = socket.gethostbyname(self.host)
        return host, port

    # Custom function: Close the session
    def closeSession(self):
        try:
            self.close()
            if (self.logLevel > 1): self._log("INFO - FTPS close() done")
        except Exception as e:
            if (self.logLevel > 0): self._log("ERROR - FTPS close() failed - " + str(e))
            raise e
        if (self.logLevel > 1): self._log("INFO - FTPS session successfully closed")

    # Private method for logs
    def _log(self, msg):
        # Be free here on how to implement your own way to redirect logs (e.g: to a console, to a file, etc.)
        print(msg)

Usage example

host = "www.myserver.com"
port = 990
user = "myUserId"
password = "myPassword"

myFtps = FTPTLS()
myFtps.logLevel = 2
myFtps.openSession(host, port, user, password)
print(myFtps.retrlines("LIST"))
myFtps.closeSession()

Output example

INFO - FTPS connect() done: 220 (vsFTPd 3.0.2)
INFO - FTPS prot_p() done: 200 PROT now Private.
INFO - FTPS login() done: 230 Login successful.
INFO - FTPS session successfully opened
-rw-------    1 ftp      ftp         86735 Mar 22 16:55 MyModel.yaml
-rw-------    1 ftp      ftp          9298 Mar 22 16:55 MyData.csv
226 Directory send OK.
INFO - FTPS close() done
INFO - FTPS session successfully closed

Answer 2

I also had the same problem with a FTP server using implicit TLS (also in passive mode). I ran into multiple errors:

450 TLS session of data connection has not resumed or the session does not match the control connection

with the proposed solutions. For the record, my solution based on @George Leslie-Waksman is:

class ImplicitFTP_TLS(ftplib.FTP_TLS):
    """
    FTP_TLS subclass that automatically wraps sockets in SSL to support implicit FTPS.
    Prefer explicit TLS whenever possible.
    """

    def __init__(self, *args, **kwargs):
        """Initialise self."""
        super().__init__(*args, **kwargs)
        self._sock = None

    @property
    def sock(self):
        """Return the socket."""
        return self._sock

    @sock.setter
    def sock(self, value):
        """When modifying the socket, ensure that it is SSL wrapped."""
        if value is not None and not isinstance(value, ssl.SSLSocket):
            value = self.context.wrap_socket(value)
        self._sock = value

    def ntransfercmd(self, cmd, rest=None):
        """Override the ntransfercmd method"""
        conn, size = ftplib.FTP.ntransfercmd(self, cmd, rest)
        conn = self.sock.context.wrap_socket(
            conn, server_hostname=self.host, session=self.sock.session
        )
        return conn, size

The novelty lies in the ntransfercmd override.

Then it is used similarly to ftplib.FTP_TLS (numerous examples in previous posts).

Overriding connect instead of using the sock property / setter works too.

Answer 3

Extending the solutions that have been proposed so far, the issue is that implicit FTPS connections need the socket to be ssl wrapped automatically, before we get a chance to call login(). A lot of the subclasses that people are proposing do this in the context of the connect method, we can more generally manage this by modifying the get/set of self.sock with a property to auto-wrap on set:

import ftplib
import ssl

class ImplicitFTP_TLS(ftplib.FTP_TLS):
    """FTP_TLS subclass that automatically wraps sockets in SSL to support implicit FTPS."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._sock = None

    @property
    def sock(self):
        """Return the socket."""
        return self._sock

    @sock.setter
    def sock(self, value):
        """When modifying the socket, ensure that it is ssl wrapped."""
        if value is not None and not isinstance(value, ssl.SSLSocket):
            value = self.context.wrap_socket(value)
        self._sock = value

Usage is essentially the same as with the standard FTP_TLS class:

ftp_client = ImplicitFTP_TLS()
ftp_client.connect(host='ftp.example.com', port=990)
ftp_client.login(user='USERNAME', passwd='PASSWORD')
ftp_client.prot_p()

Answer 4

I've worked on the same problem for half a day and finally figured it out.

For the implicit FTP TLS/SSL(defualt port 990), our client program must build a TLS/SSL connection right after the socket is created. But python's class FTP_TLS doesn't reload the connect function from class FTP. We need to fix it:

class tyFTP(ftplib.FTP_TLS):
  def __init__(self,
               host='',
               user='',
               passwd='',
               acct='',
               keyfile=None,
               certfile=None,
               timeout=60):

    ftplib.FTP_TLS.__init__(self,
                            host=host,
                            user=user,
                            passwd=passwd,
                            acct=acct,
                            keyfile=keyfile,
                            certfile=certfile,
                            timeout=timeout)

  def connect(self, host='', port=0, timeout=-999):
    """Connect to host.  Arguments are:
    - host: hostname to connect to (string, default previous host)
    - port: port to connect to (integer, default previous port)
    """
    if host != '':
        self.host = host
    if port > 0:
        self.port = port
    if timeout != -999:
        self.timeout = timeout
    try:
        self.sock = socket.create_connection((self.host, self.port), self.timeout)
        self.af = self.sock.family
        # add this line!!!
        self.sock = ssl.wrap_socket(self.sock,
                                    self.keyfile,
                                    self.certfile,
                                    ssl_version=ssl.PROTOCOL_TLSv1)
        # add end
        self.file = self.sock.makefile('rb')
        self.welcome = self.getresp()
    except Exception as e:
        print(e)
    return self.welcome

This derived class reloads the connect function and builds a wrapper around the socket to TLS. After you successfully connect and login to FTP server, you need to call: FTP_TLS.prot_p() before executing any FTP command!

Hope this will help ^_^

Answer 5

I know this thread is quite old and ftp is not as popular as it once was but anyways, in case it helps anybody I'd like to make an additional contribution. I ran into a similar situation trying to connect to ftp server using IMPLICIT (Port 990) ftps in PASSIVE mode. In that situation the server, after the initial connection is negotiated, usually provides a new host IP address and port, likely different from the ones that were used to make the initial connection, over which the actual data transfers are supposed to happen. No big deal, ftps clients, including python, can handle this, only this particular server was providing a non-routable (likely internal to the firewall) IP address. I noticed that FileZilla connected no problem, but python ftplib could not. Then I ran into this thread:

How to replace a non routable IP address with server address on ftplib

which clued me in. Using Grzegorz Wierzowiecki methodology I extended the method alluded to in that thread and came up with this, which solved my problem.

import ftplib, os, sys
import socket
import ssl
FTPS_OBJ = ftplib.FTP_TLS



def conn_i_ftps(FTP_Site, Login_Name, Login_Password):
    print "Starting IMPLICIT ftp_tls..."
    ftps = tyFTP()
    print ftps.connect(host=FTP_Site, port=990, timeout=120)
    ftps.prot_p()
    ftps.login(user=Login_Name, passwd=Login_Password)
    print "Logged In"
    ftps.retrlines('LIST')
    # return ftps


class tyFTP(FTPS_OBJ):
    def __init__(self, host='', user='', passwd='', acct='', keyfile=None, certfile=None, timeout=60):
        FTPS_OBJ.__init__(self, host, user, passwd, acct, keyfile, certfile, timeout)

    def connect(self, host='', port=0, timeout=-999):
        if host != '':
            self.host = host
        if port > 0:
            self.port = port
        if timeout != -999:
            self.timeout = timeout

        try:
            self.sock = socket.create_connection((self.host, self.port), self.timeout)
            self.af = self.sock.family
            self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)
            self.file = self.sock.makefile('rb')
            self.welcome = self.getresp()
        except Exception as e:
            print e
        return self.welcome

    def makepasv(self):
        print port #<---Show passively assigned port
        print host #<---Show the non-routable, passively assigned IP
        host, port = FTPS_OBJ.makepasv(self)
        host = socket.gethostbyname(self.host) #<---- This changes the host back to the original IP that was used for the connection
        print 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
        print host #<----Showing the original IP
        return host, port

Then the code gets called like this

FTP_Site       =  "ftp.someserver.com"
Login_Name     =  "some_name"
Login_Password =  "some_passwd"

conn_i_ftps(FTP_Site, Login_Name, Login_Password)

I suppose one could wrap the lines where the host gets changed back with an IF statement identifying non-routable addresses, sort of like this:

if host.split(".")[0] in (10, 192, 172):
     host = socket.gethostbyname(self.host)
     .
     .
     .

Answer 6

The answer by NERV and the sample by Brad Decker was really helpful. Kudos to them. They saved me hours.

Unfortunetely, initially it didn't work for me.

In my case, the connection just worked once I removed the ssl_version parameter from the ssl.wrap_socket method. Also, to send any command to the server, I had to overwrite the ntransfercmd method from the FTP_TLS class and remove the ssl_version parameter there too.

That's the code that worked for me:

from ftplib import FTP_TLS, FTP
import socket
import ssl

class IMPLICIT_FTP_TLS(FTP_TLS):
    def __init__(self, host='', user='', passwd='', acct='', keyfile=None,
        certfile=None, timeout=60):
        FTP_TLS.__init__(self, host, user, passwd, acct, keyfile, certfile, timeout)

    def connect(self, host='', port=0, timeout=-999):
        '''Connect to host.  Arguments are:
        - host: hostname to connect to (string, default previous host)
        - port: port to connect to (integer, default previous port)
        '''
        if host != '':
            self.host = host
        if port > 0:
            self.port = port
        if timeout != -999:
            self.timeout = timeout
        try:
            self.sock = socket.create_connection((self.host, self.port), self.timeout)
            self.af = self.sock.family
            self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)
            self.file = self.sock.makefile('rb')
            self.welcome = self.getresp()
        except Exception as e:
            print (e)
        return self.welcome

    def ntransfercmd(self, cmd, rest=None):
        conn, size = FTP.ntransfercmd(self, cmd, rest)
        if self._prot_p:
            conn = ssl.wrap_socket(conn, self.keyfile, self.certfile)
        return conn, size

And the obligatory sample:

>>> ftps = IMPLICIT_FTP_TLS()
>>> ftps.connect(host='your.ftp.host', port=990)
>>> ftps.login(user="your_user", passwd="your_passwd")
>>> ftps.prot_p()
>>> ftps.retrlines('LIST')

Answer 7

Extending upon NERV's response - which helped me immensely, here is how I was able to solve my problem with a implicit TLS connection on port 990 requiring authentication.

Filename: ImplicitTLS.py

from ftplib import FTP_TLS
import socket
import ssl

class tyFTP(FTP_TLS):
    def __init__(self, host='', user='', passwd='', acct='', keyfile=None, certfile=None, timeout=60):
        FTP_TLS.__init__(self, host, user, passwd, acct, keyfile, certfile, timeout)
    def connect(self, host='', port=0, timeout=-999):
        if host != '':
            self.host = host
        if port > 0:
            self.port = port
        if timeout != -999:
            self.timeout = timeout

        try: 
            self.sock = socket.create_connection((self.host, self.port), self.timeout)
            self.af = self.sock.family
            self.sock = ssl.wrap_socket(self.sock, self.keyfile, self.certfile, ssl_version=ssl.PROTOCOL_TLSv1)
            self.file = self.sock.makefile('rb')
            self.welcome = self.getresp()
        except Exception as e:
            print e
        return self.welcome

Then from my main application I did this:

from ImplicityTLS import tyFTP
server = tyFTP()
server.connect(host="xxxxx", port=990)
server.login(user="yyyy", passwd="fffff")
server.prot_p()

And that was it, i was able to download files, etc. Props go to NERV for the original answer.