Python Twisted: Is it possible to reload certificates on the fly?

Question

I have a simple web server which serves content over HTTPS:

sslContext = ssl.DefaultOpenSSLContextFactory(
  '/home/user/certs/letsencrypt-privkey.pem',
  '/home/user/certs/letsencrypt-fullchain.pem',
)
reactor.listenSSL(
  port=https_server_port,
  factory=website_factory,
  contextFactory=sslContext,
  interface=https_server_interface
)
do_print(bcolors.YELLOW + 'server.py | running https server on ' + https_server_interface + ':' + str(https_server_port) + bcolors.END)

Is it possible to reload the certificates on the fly (for example by calling a path like https://example.com/server/reload-certificates and having it execute some code) or what do I need to do in order to get it done?

I want to avoid restarting the Python process.

Answer 1

It is possible.

reactor.listenSSL returns a twisted.internet.tcp.Port instance which you can store somewhere accessible like in the website resource of your server, so that you can later access it:

website_resource = Website()

website_factory = server.Site(website_resource)

website_resource.sslPort = reactor.listenSSL( # <---
  port=https_server_port,
  factory=website_factory,
  contextFactory=sslContext,
  interface=https_server_interface
)

then later in your http handler (render function) you can execute the following:

if request.path == b'/server/reload-certificates':
  request.setHeader("connection", "close")
  self.sslPort.connectionLost(reason=None)
  self.sslPort.stopListening()
  self.sslListen()
  return b'ok'

where self.sslListen is the initial setup code:

website_resource = Website()

website_factory = server.Site(website_resource)

def sslListen():
  sslContext = ssl.DefaultOpenSSLContextFactory(
    '/home/user/certs/letsencrypt-privkey.pem',
    '/home/user/certs/letsencrypt-fullchain.pem',
  )
  website_resource.sslPort = reactor.listenSSL(
    port=https_server_port,
    factory=website_factory,
    contextFactory=sslContext,
    interface=https_server_interface
  )

website_resource.sslListen = sslListen # <---
sslListen() # invoke once initially

# ...

reactor.run()

Notice that request.setHeader("connection", "close") is optional. It indicates the browser that it should close the connection and not reuse it for the next fetch to the server (HTTP/1.1 connections usually are kept open for at least 30 seconds in order to be reused).

If the connection: close header is not sent, then everything will still work, the connection will still be active and usable, but it will still be using the old certificate, which should be no problem if you're just reloading the certificates to refresh them after certbot updated them. New connections from other browsers will start using the new certificates immediately.

Answer 2

It is possible in several ways. Daniel F's answer is pretty good and shows a good, general technique for reconfiguring your server on the fly.

Here are a couple more techniques that are more specific to TLS support in Twisted.

First, you could reload the OpenSSL "context" object from the DefaultOpenSSLContextFactory instance. When it comes time to reload the certificates, run:

sslContext._context = None
sslContext.cacheContext()

The cacheContext call will create a new OpenSSL context, re-reading the certificate files in the process. This does have the downside of relying on a private interface (_context) and its interaction with a not-really-that-public interface (cacheContext).

You could also implement your own version of DefaultOpenSSLContextFactory so that you don't have to rely on these things. DefaultOpenSSLContextFactory doesn't really do much. Here's a copy/paste/edit that removes the caching behavior entirely:

class DefaultOpenSSLContextFactory(ContextFactory):
    """
    L{DefaultOpenSSLContextFactory} is a factory for server-side SSL context
    objects.  These objects define certain parameters related to SSL
    handshakes and the subsequent connection.
    """
    _context = None

    def __init__(self, privateKeyFileName, certificateFileName,
                 sslmethod=SSL.SSLv23_METHOD, _contextFactory=SSL.Context):
        """
        @param privateKeyFileName: Name of a file containing a private key
        @param certificateFileName: Name of a file containing a certificate
        @param sslmethod: The SSL method to use
        """
        self.privateKeyFileName = privateKeyFileName
        self.certificateFileName = certificateFileName
        self.sslmethod = sslmethod
        self._contextFactory = _contextFactory

    def getContext(self):
        """
        Return an SSL context.
        """
        ctx = self._contextFactory(self.sslmethod)
        # Disallow SSLv2!  It's insecure!  SSLv3 has been around since
        # 1996.  It's time to move on.
        ctx.set_options(SSL.OP_NO_SSLv2)
        ctx.use_certificate_file(self.certificateFileName)
        ctx.use_privatekey_file(self.privateKeyFileName)

Of course, this reloads the certificate files for every single connection which may be undesirable. You could add your own caching logic back in, with a control interface that fits into your certificate refresh system. This also has the downside that DefaultOpenSSLContextFactory is not really a very good SSL context factory to begin with. It doesn't follow current best practices for TLS configuration.

So you probably really want to use twisted.internet.ssl.CertificateOptions instead. This has a similar _context cache that you could clear out:

sslContext = CertificateOptions(...) # Or PrivateCertificate(...).options(...)
...
sslContext._context = None

It will regenerate the context automatically when it finds that it is None so at least you don't have to call cacheContext this way. But again you're relying on a private interface.

Another technique that's more similar to Daniel F's suggestion is to provide a new factory for the already listening socket. This avoids the brief interruption in service that comes between stopListening and listenSSL. This would be something like:

from twisted.protocols.tls import TLSMemoryBIOFactory

# DefaultOpenSSLContextFactory or CertificateOptions or whatever
newContextFactory = ... 
tlsWebsiteFactory = TLSMemoryBIOFactory(
    newContextFactory,
    isClient=False,
    websiteFactory,
)

listeningPortFileno = sslPort.fileno()
websiteFactory.sslPort.stopReading()
websiteFactory.sslPort = reactor.adoptStreamPort(
    listeningPortFileno, 
    AF_INET, 
    tlsWebsiteFactory,
)

This basically just has the reactor stop servicing the old sslPort with its outdated configuration and tells it to start servicing events for that port's underlying socket on a new factory. In this approach, you have to drop down to the slightly lower level TLS interface since you can't adopt a "TLS port" since there is no such thing. Instead, you adopt the TCP port and apply the necessary TLS wrapping yourself (this is just what listenSSL is doing for you under the hood).

Note this approach is a little more limited than the others since not all reactors provide the fileno or adoptStreamPort methods. You can test for the interfaces the various objects provide if you want to use this where it's supported and degrade gracefully elsewhere.

Also note that since TLSMemoryBIOFactory is how it always works under the hood anyway, you could also twiddle its private interface, if you have a reference to it:

tlsMemoryBIOFactory._connectionCreator = IOpenSSLServerConnectionCreator(
    newContextFactory,
)

and it will begin using that for new connections. But, again, private...