SSL for JMX with RMI

Question

We have a Java application which has had a JConsole connection with password authentication for a while. In improving the security of this, we are trying to encrypt the connection made from JConsole to the application.

Up until now, we have launched our application with the following launch command:

java -Dcom.sun.management.jmxremote \
     -Dcom.sun.management.jmxremote.port=1099 \
     -Dcom.sun.management.jmxremote.rmi.port=1099 \
     -Dcom.sun.management.jmxremote.authenticate=true \
     -Dcom.sun.management.jmxremote.password.file=jmx.password \
     -Dcom.sun.management.jmxremote.access.file=jmx.access \
     -Dcom.sun.management.jmxremote.ssl=false
     -jar MyApplication.jar

With this, we can flawlessly access the JMX methods of MyApplication via both JConsole, jmxterm, and other Java applications. In JConsole and jmxterm, we can use both hostname:1099 and service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi without issues. From the Java applications, we always use service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi, again without issues. Our application has no code-based setup of the JMX endpoint (we exposes some methods and attributes, but we did not touch the registry and socket factories).

Now we are trying to set up SSL between our application, and all other parties, following www.cleantutorials.com/jconsole/jconsole-ssl-with-password-authentication. Doing this, we have a keystore and truststore for both MyApplication and whoever the client connection to the JMX methods is. We use

java -Dcom.sun.management.jmxremote \
     -Dcom.sun.management.jmxremote.port=1099 \
     -Dcom.sun.management.jmxremote.rmi.port=1099 \
     -Dcom.sun.management.jmxremote.authenticate=true \
     -Dcom.sun.management.jmxremote.password.file=jmx.password \
     -Dcom.sun.management.jmxremote.access.file=jmx.access \
     -Dcom.sun.management.jmxremote.ssl=true \
     -Dcom.sun.management.jmxremote.ssl.need.client.auth=true \
     -Dcom.sun.management.jmxremote.registry.ssl=true \
     -Djava.rmi.server.hostname=hostname \
     -Djavax.net.ssl.keyStore=server-jmx-keystore \
     -Djavax.net.ssl.keyStorePassword=password \
     -Djavax.net.ssl.trustStore=server-jmx-truststore \
     -Djavax.net.ssl.trustStorePassword=password \
     -jar MyApplication.jar

After this, almost all our connections fail. The only one succeeding, is via JConsole (adding the client keystore and truststores to the launch config), and only using hostname:1099. Using the address service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi no longer works, not via JConsole, not via jmxterm, and not via other applications.

We have tried about any combination of launch settings we could think of, but nothing that we find anywhere seems to work. The error we see when trying to connect from e.g. jmxterm is:

java.rmi.ConnectIOException: non-JRMP server at remote endpoint

(I can provide the full stack if needed).

We're a bit at a loss on how to continue, what we can do to make all connections that used to work, now work. What should we do to enable connecting with service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi-like connection strings via SSL?

If relevant, this application is using OpenJDK 11.0.5, other applications where we might need this run on OpenJDK 8.

Edit

Debugging both the JConsole client and the backend side, it seems that the protocol that the client is trying to establish is not known in the SSL context. On the backend, we have the following error:

javax.net.ssl|DEBUG|20|RMI TCP Connection(1)|2021-12-28 10:04:04.265 CET|null:-1|Raw read (
  0000: 4A 52 4D 49 00
                JRMI.
)
javax.net.ssl|ERROR|20|RMI TCP Connection(1)|2021-12-28 10:04:04.267 CET|null:-1|Fatal (UNEXPECTED_MESSAGE): Unsupported or unrecognized SSL message (
"throwable" : {
  javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
          at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source)
          at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source)
          at java.base/java.io.BufferedInputStream.fill(Unknown Source)
          at java.base/java.io.BufferedInputStream.read(Unknown Source)
          at java.base/java.io.DataInputStream.readInt(Unknown Source)
          at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
          at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(Unknown Source)
          at java.base/java.security.AccessController.doPrivileged(Native Method)
          at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.base/java.lang.Thread.run(Unknown Source)}
)

After which the backend closes the connection.

Based on some tutorials online, it should be possible to get the SSL connection working using the service-based URL, but we can't get it to work.

Answer 1

After a long search, with a lot of debugging, trial and error, we came to the conclusion that there is no out-of-the-box solution in Spring (Boot) to enable SSL with an RMI registry and a JMX connection server. This had to be configured manually. We used the following Spring configuration class that did the trick:

@Configuration
@EnableMBeanExport
public class JMXConfig {

    private static final Log LOG = LogFactory.getLog(JMXConfig .class);

    @Value("${jmx.registry.port:1098}")
    private Integer registryPort;
    @Value("${jmx.rmi.port:1099}")
    private Integer rmiPort;

    @Bean
    public RmiRegistryFactoryBean rmiRegistry() {
        final RmiRegistryFactoryBean rmiRegistryFactoryBean = new RmiRegistryFactoryBean();
        rmiRegistryFactoryBean.setPort(rmiPort);
        rmiRegistryFactoryBean.setAlwaysCreate(true);

        LOG.info("Creating RMI registry on port " + rmiRegistryFactoryBean.getPort());
        return rmiRegistryFactoryBean;
    }

    @Bean
    @DependsOn("rmiRegistry")
    public ConnectorServerFactoryBean connectorServerFactoryBean() throws MalformedObjectNameException {
        String rmiHost = getHost();
        String serviceURL = serviceURL(rmiHost);
        LOG.info("Creating JMX connection for URL " + serviceURL);

        final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
        connectorServerFactoryBean.setObjectName("connector:name=rmi");
        connectorServerFactoryBean.setEnvironmentMap(createRmiEnvironment(rmiHost));
        connectorServerFactoryBean.setServiceUrl(serviceURL);
        return connectorServerFactoryBean;
    }

    private String getHost() {
        try {
            InetAddress localMachine = InetAddress.getLocalHost();
            return localMachine.getCanonicalHostName();
        } catch (UnknownHostException e) {
            LOG.warn("Unable to get hostname, using localhost", e);
            return "localhost";
        }
    }

    private String serviceURL(String rmiHost) {
        return format("service:jmx:rmi://%s:%s/jndi/rmi://%s:%s/jmxrmi", rmiHost, registryPort, rmiHost, rmiPort);
    }

    private Map<String, Object> createRmiEnvironment(String rmiHost) {
        final Map<String, Object> rmiEnvironment = new HashMap<>();
        rmiEnvironment.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.rmi.registry.RegistryContextFactory");
        rmiEnvironment.put(Context.PROVIDER_URL, "rmi://" + rmiHost + ":" + rmiPort);
        rmiEnvironment.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, new SslRMIClientSocketFactory());
        rmiEnvironment.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE, new SslRMIServerSocketFactory());
        return rmiEnvironment;
    }
}

This enables SSL using the connection details service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi. To make it work, you need to add a keystore/password to your backend, and a truststore/password to your frontend (as in the tutorial).

Answer 2

Hm, I'm not sure but if I read this page from CERN correctly, and I quote:

in case you don't use SSL the RMI port and the JMX port can be the same

it follows that if you want to use SSL/TLS you thus need to use 2 different ports on the server side. (This is confirmed by this comment stating: "SSL doesn't do port sharing").

Also you may want to consider using https://jolokia.org/ as HTTP+JSON alternative for all the crazyness of RMI networking, which also provides a Java client e.g. for JConsole, see the bottom of: https://jolokia.org/reference/html/clients.html for an example.

Answer 3

The thrown javax.net.ssl.SSLException: Unsupported or unrecognized SSL message is an indicator that there is plaintext instead of SSL communication. This asumption is verified by the fact that there are no SSL handshake messages (at least CLIENT_HELLO) in the debug output. So it seems to be clear that th client doesn't use SSLSocket for communication.

One very simple example for SSLSocket based communication can be found here in Oracle documentation, but there are lot more available.

Answer 4

TLS handshakes are obviously failing. Without knowing the contents of your truststore (a couple of self-signed certificates?), or the ability to inspect the certificates in exchange, it's most likely because cacerts, the default Java truststore that contains common public root certificates, is not being loaded anymore now that you're specifying your own truststore.

As a confidence check, you can import the contents of cacerts into a copy of your truststore and retry. (See keytool's importkeystore.)