How to replace the static keystore with a dynamic one for a spring-boot-application

Question

I want to add/replace SSL certificates dynamically to my spring boot (tomcat) application without the need to restart it. I have a long way to go still, but currently I am stuck with a javax.crypto.BadPaddingException and don't know why.

So here is what I am trying to do.

First, I am defining my own TomcatServletWebServerFactoryin order to set a SslStoreProvider.

@Component
public class PathWatchingTomcatFactory extends TomcatServletWebServerFactory {
  public PathWatchingTomcatFactory(PathWatchingSslStoreProvider pathWatchingSslStoreProvider) {
    setSslStoreProvider(pathWatchingSslStoreProvider);
  }
}

My PathWatchingSslStoreProvider provides a PathMatchingKeyStore.

@Component
public class PathWatchingSslStoreProvider implements SslStoreProvider {
  private final PathWatchingKeyStore pathWatchingKeyStore;

  public PathWatchingSslStoreProvider(PathWatchingKeyStore pathWatchingKeyStore) {
    this.pathWatchingKeyStore = pathWatchingKeyStore;
  }

  @Override
  public KeyStore getKeyStore() throws Exception {
    return pathWatchingKeyStore;
  }
}

The PathWatchingKeyStore seems only necessary in order to provide a service provider interface to it.

@Component
public class PathWatchingKeyStore extends KeyStore {
  protected PathWatchingKeyStore(
    PathWatchingKeyStoreSpi pathWatchingKeyStoreSpi,
    DynamicProvider provider)
  {
    super(pathWatchingKeyStoreSpi, provider, KeyStore.getDefaultType());

    initialize();
  }

  private void initialize() {
    // Loading a keystore marks it internally as initialized and only
    // initialized keystores work properly. Unfortunately
    // nobody initializes this keystore. So we have to do it
    // ourselves.
    //
    // Internally the keystore will delegate loading to the
    // KeyStoreSpi, which, in our case is the PathWatchingKeyStoreSpi.
    try {
      load(null, null);
    }
    catch (Exception e) {
      e.printStackTrace();
    }
  }
}

Now, on startup, the keystore will be loaded. And because I provide a SslStoreProvider my keystore will be loaded by the SslStoreProviderUrlStreamHandlerFactory by requesting my PathWatchingKeyStoreSpi to store its keystore into a ByteArrayOutputStream whose content is finally copied into the InputStream that is used to load an internally used keystore.

In the following code snippet you can see how I try to write the contents of an already existing keystore. No dynamic at all right now. I only want to see if the spring boot application starts with all these custom classes in place. But it doesn't.

@Component
public class PathWatchingKeyStoreSpi extends KeyStoreSpi {
  private static final Logger LOGGER = LoggerFactory.getLogger(PathWatchingKeyStoreSpi.class);

  private final Path keyStoreLocation;

  public PathWatchingKeyStoreSpi(@Value("${server.ssl.key-store}") Path keyStoreLocation) {
    super();

    this.keyStoreLocation = keyStoreLocation;
  }

  @Override
  public void engineStore(OutputStream stream, char[] password) throws IOException, NoSuchAlgorithmException, CertificateException {
    try {
      final KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
      keyStore.load(new FileInputStream(keyStoreLocation.toString()), "secret".toCharArray());

      // Password must be empty because the SslConnectorCustomizer sets the keystore
      // password used by the tomcat to the empty string if the SslStoreProvider
      // returns a keystore. And because that is what we wanted to do in the first place,
      // providing a dynamic keystore, this is what we have to do.
      keyStore.store(stream, "".toCharArray());
    }
    catch (Exception e) {
      e.printStackTrace();
    }
  }
}

I can see that the keystore is loaded but as soon as the SSLUtilBase tries to read the key from that store, it throws a BadPaddingException:

Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
    at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:975) ~[na:na]
    at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1056) ~[na:na]
    at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853) ~[na:na]
    at java.base/com.sun.crypto.provider.PKCS12PBECipherCore.implDoFinal(PKCS12PBECipherCore.java:408) ~[na:na]
    at java.base/com.sun.crypto.provider.PKCS12PBECipherCore$PBEWithSHA1AndDESede.engineDoFinal(PKCS12PBECipherCore.java:440) ~[na:na]
    at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202) ~[na:na]
    at java.base/sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:406) ~[na:na]
    at java.base/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:302) ~[na:na]
    at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:400) ~[na:na]
    ... 25 common frames omitted

I created the static keystore I am using here as follows:

keytool -genkey -alias tomcat -keyalg RSA

First of all, is the direction I am going to solve my problem promising? Or am I totally wrong? I first tried to only inject my own X509ExtendedKeyManager. I could see in the debugger that it is the key manager that is asked for a certificate for an incoming request but nonetheless the tomcat endpoint seems to be initialized with a keystore without the manager being involved.

Has anybody ever tried to implement and use a dynamic keystore/trustore for a spring boot application using tomcat as servelt container?

Any help is welcome. Tobias

Answer 1

Ok, I don't know if this is final solution but right now it seems a lot more promising (and less complex) then my first way described above.

Again it all starts with the TomcatServletWebServerFactory. But this time I set a completely new JSSEImplementation:

@Component
public class PathWatchingTomcatFactory extends TomcatServletWebServerFactory {
  private final Path keysLocation;

  public PathWatchingTomcatFactory(@Value("${tobias.spring.ssl.keys-location}")Path keysLocation) {
    this.keysLocation = requireNonNull(keysLocation);
  }

  @Override
  protected void customizeConnector(Connector connector) {
    super.customizeConnector(connector);

    connector.setProperty("sslImplementationName", DynamicSslImplementation.class.getName());
    System.setProperty("tobias.spring.ssl.keys-location", keysLocation.toUri().toString());
  }
}

The implementation class is very simple. It only has to provide a custom SSLUtil instance.

public class DynamicSslImplementation extends JSSEImplementation {
  public DynamicSslImplementation() {
    super();
  }

  @Override
  public SSLUtil getSSLUtil(SSLHostConfigCertificate certificate) {
    return new DynamicSslUtil(certificate);
  }
}

And the SSLUtil instance provides my own PathWatchingKeyManager, which will return keys from a certain directory.

public class DynamicSslUtil extends JSSEUtil {
  DynamicSslUtil(SSLHostConfigCertificate certificate) {
    super(certificate);
  }

  @Override
  public KeyManager[] getKeyManagers() {
    return new KeyManager[]{new DynamicKeyManager()};
  }
}

The server.ssl.key-storeproperty must be set to NONE.

This seems to work. The spring boot applications starts running without failures and the DynamicKeyManager is asked for a certificate for a https request.

If this will work indeed, I will post the complete solution here.