Python Explicit FTP doesn't negotiate TLS

Question

I'm trying to transfer a file to a host over FTP TLS explicit, but don't know how to handle the certificate. A WinSCP log of the transaction shows that the TLS negotiation is handled and the certificate is verified. But my Python script fails to do this. I know the certificate fingerprint and the cipher, but don't know how to implement it.

My script:

import ftplib
import ssl


def main():
    ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1_2)
    ftps = ftplib.FTP_TLS(context=ctx)

    try:
        ftps.set_debuglevel(2)
        print(ftps.connect(host))
        ftps.auth()
        print(ftps.login(username,password))
        print("1")
        ftps.prot_p()
        print("2")
        #ftps.cwd('/')
        print("3")
        print (ftps.retrlines('LIST'))
        print("4")
        ftps.storbinary("STOR test.csv", open('C:\\test.csv', 'rb'))
        print("5")
    except Exception as ex:
        print("error: ")
        print(ex)

    ftps.close()

    input('Hit <ENTER> to close...')


if __name__ == "__main__":
    main()

Here is my output, the last line is the Python error:

*get* '220 Microsoft FTP Service\n'
*resp* '220 Microsoft FTP Service'
220 Microsoft FTP Service
*cmd* 'AUTH TLS'
*put* 'AUTH TLS\r\n'
*get* '234 AUTH command ok. Expecting TLS Negotiation.\n'
*resp* '234 AUTH command ok. Expecting TLS Negotiation.'
*cmd* 'USER xxx'
*put* 'USER xxx\r\n'
*get* '331 Password required\n'
*resp* '331 Password required'
*cmd* 'PASS ********'
*put* 'PASS ********\r\n'
*get* '230 User logged in.\n'
*resp* '230 User logged in.'
230 User logged in.
1
*cmd* 'PBSZ 0'
*put* 'PBSZ 0\r\n'
*get* '200 PBSZ command successful.\n'
*resp* '200 PBSZ command successful.'
*cmd* 'PROT P'
*put* 'PROT P\r\n'
*get* '200 PROT command successful.\n'
*resp* '200 PROT command successful.'
2
3
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A.\n'
*resp* '200 Type set to A.'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (x,x,x,x,4,4).\n'
*resp* '227 Entering Passive Mode (x,x,x,x,4,4).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '125 Data connection already open; Transfer starting.\n'
*resp* '125 Data connection already open; Transfer starting.'
*get* '226 Transfer complete.\n'
*resp* '226 Transfer complete.'
226 Transfer complete.
4
*cmd* 'TYPE I'
*put* 'TYPE I\r\n'
*get* '200 Type set to I.\n'
*resp* '200 Type set to I.'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (x,x,x,x,4,3).\n'
*resp* '227 Entering Passive Mode (x,x,x,x,4,3).'
*cmd* 'STOR test.csv'
*put* 'STOR test.csv\r\n'
*get* '125 Data connection already open; Transfer starting.\n'
*resp* '125 Data connection already open; Transfer starting.'
error: 
The read operation timed out

Answer 1

The issue turned out to be with the unwrapping of the socket after the file is transmitted. More specifically, the shutdown() method appears to wait for the server to see if it's okay to close the socket. In this case the server doesn't answer. The file does indeed get transmitted, but the socket issue throws an error before a "File transfer complete" response can be received.

It seems to go like this (sending file to server): 1. Establish FTP TLS 2. Client issues STOR command 3. A secure socket is created 4. The file is transferred 5. The socket is unwrapped and disposed of. 6. The "transfer complete" response is received from the server.

But an issue with step 5 prevents 6 from happening.

I'm not sure if this is a bug or not. A little bit of research shows that sockets can be closed instead of shutdown and that servers may not respond to socket shutdown requests.

My solution was to modify my local python libraries to deal with it.

Answer 2

A WinSCP log of the transaction shows that the TLS negotiation is handled and the certificate is verified. But my Python script fails to do this. I know the certificate fingerprint and the cipher, but don't know how to implement it.

Your Python script does not fail in the TLS negotiation as you assume.

An upgrade from a plain connection to TLS is requested by the client with the AUTH TLS command which is accepted by the server which can be seen in the log:

*put* 'AUTH TLS\r\n'
*get* '234 AUTH command ok. Expecting TLS Negotiation.\n'

After that the TLS negotiation (i.e. TLS handshake) is done. If the handshake would fail the client would abort. But the handshake succeeds so that the client can continue with more command which also get accepted by the server:

*put* 'USER xxx\r\n'
*get* '331 Password required\n'

Thus, there is no problem in the TLS negotiation. It is also not a problem with the data transfer itself which can be seen in that the client succeeds in transferring data from the server, i.e. the directory listing:

*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (x,x,x,x,4,4).\n'
*put* 'LIST\r\n'
*get* '125 Data connection already open; Transfer starting.\n'
*get* '226 Transfer complete.\n'

It only fails when transferring a file to the server because the server does not reply within the expected time:

*put* 'STOR test.csv\r\n'
*get* '125 Data connection already open; Transfer starting.\n'
error: 
The read operation timed out

It is unclear why the server fails to respond. One possible reason might be that the file needs to be processed by some application (like an antivirus) before the server will respond with success and that this processing takes too long.

Answer 3

Comment:
ssl.create_default_context() results in an error: "[SSL: CERTIFICATE_VERIFY_FAILED]"

---

ssl.create_default_context(purpose=Purpose.SERVER_AUTH, cafile=None, capath=None, cadata=None)
Return a new SSLContext object with default settings for the given purpose.

cafile, capath, cadata represent optional CA certificates to trust for certificate verification, as in SSLContext.load_verify_locations(). If all three are None, this function can choose to trust the system’s default CA certificates instead.

Use as follows:

ctx = ssl.create_default_context(Purpose.CLIENT_AUTH)

---

Question: don't know how to handle the certificate

Use ctx = ssl.load_cert_chain(certfile, keyfile=None, password=None),
you already have a context=ctx.

class ftplib.FTP_TLS(host=”, user=”, passwd=”, acct=”, keyfile=None, certfile=None, context=None, timeout=None, source_address=None)

A FTP subclass which adds TLS support to FTP as described in RFC 4217.
context= is a ssl.SSLContext object which allows bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure.

keyfile= and certfile= are a legacy alternative to context – they can point to PEM-formatted private key and certificate chain files (respectively) for the SSL connection.

---

Deprecated since version 3.6:
keyfile and certfile are deprecated in favor of context. Please use ssl.SSLContext.load_cert_chain() instead, or let ssl.create_default_context() select the system’s trusted CA certificates for you.