Unable do client authentication with PKCS#1 private key with java keyStore

Question

I have public key and private key (PKCS#1 : i.e having "BEGIN RSA PRIVATE KEY") as separate String inputs. These keys are created using below commands:

openssl req -new -x509 -keyout ca-key.pem -out ca-cert.pem -days 365 -passout pass:abcd123 -subj "/C=IN/ST=KAD/L=CAT/O=MIN/OU=OPQ/CN=*.xyz.com/emailAddress=xxx.xxxx@xyz.com"
openssl genrsa -out client_private.key 2048
openssl req -new -sha256 -key client_private.key -subj "/C=IN/ST=KAD/L=CAT/O=MIN/OU=OPQ/CN=client.xyx.com" -out client.csr
openssl x509 -req -in client.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out client.crt -days 365 -sha256
openssl x509 -in client.crt -out client.pem -outform PEM

where the content of client.csr is provided as public key string and the content of client_private.key is provided as private key string

Now I have to create java keystore, add these keys and export as JKS file pragmatically. I have gone through multiple links and tried bouncy castle example but I am getting below error.

javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2

I am writing a kafka client with SSL and client auth enabled and sever is set will all required parameters. creating trust store and keystore commandline works fine. But using java program I am unable to do client auth (SSL works fine). Below is code snip:

public void initClientAuthStore(String certString, String keyString)
        throws GeneralSecurityException, IOException {
    clientAuthStore = KeyStore.getInstance(getType());
    clientAuthStore.load(null, getPassword().toCharArray());

    Certificate clientCert = CertificateHandler.getCertificate(certString);
    Certificate[] certificateChain = {trustStoreCertificate, clientCert};

    log.debug("initClientAuthStore: Certificate is created");
    // keyString.getBytes(StandardCharsets.UTF_8);
    //    clientAuthStore.setKeyEntry(
    //        keyStoreAlias, getPrivateKey(keyString), getPassword().toCharArray(), certificate);
    PrivateKey privateKey = getPrivateKey(keyString);
    clientAuthStore.setKeyEntry(
            keyStoreAlias,
            privateKey,
            getPassword().toCharArray(),
            certificateChain);

    log.debug("initClientAuthStore: setting certificate and private key is success.");
}

public PrivateKey getPrivateKey(String key) throws IOException {
    PemObject privateKeyObject;
    PemReader pemReader =
            new PemReader(
                    new InputStreamReader(
                            new ByteArrayInputStream(key.getBytes(Charset.forName("UTF-8"))),
                            StandardCharsets.UTF_8));
    privateKeyObject = pemReader.readPemObject();
    RSAPrivateCrtKeyParameters privateKeyParameter;
    if (privateKeyObject.getType().endsWith("RSA PRIVATE KEY")) {
        log.info("PRIVATE KEY TYPE: pkcs#1");
        // PKCS#1 key
        RSAPrivateKey rsa = RSAPrivateKey.getInstance(privateKeyObject.getContent());
        privateKeyParameter =
                new RSAPrivateCrtKeyParameters(
                        rsa.getModulus(),
                        rsa.getPublicExponent(),
                        rsa.getPrivateExponent(),
                        rsa.getPrime1(),
                        rsa.getPrime2(),
                        rsa.getExponent1(),
                        rsa.getExponent2(),
                        rsa.getCoefficient());
    } else if (privateKeyObject.getType().endsWith("PRIVATE KEY")) {
        log.info("PRIVATE KEY TYPE: pkcs#8");
        // PKCS#8 key
        privateKeyParameter =
                (RSAPrivateCrtKeyParameters) PrivateKeyFactory.createKey(privateKeyObject.getContent());
    } else {
        throw new RuntimeException("Unsupported key type: " + privateKeyObject.getType());
    }

    JcaPEMKeyConverter jcaPemKeyConverter = new JcaPEMKeyConverter();
    PrivateKey privateKey =
            jcaPemKeyConverter.getPrivateKey(
                    PrivateKeyInfoFactory.createPrivateKeyInfo(privateKeyParameter));
    log.info("PRIVATE KEY generated. {}", privateKey.getFormat());
    return privateKey;
}

Getting below error:

javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2

Not sure what is the wrong and if I am missing any encode/decode etc.. I get PKCS#1 private key as String input and I have to use bouncy castle to convert to PKCS#8 and generate privatekey and add to store. Note: private key is content of client_private.key

Answer 1

(Not an answer but too much for comments.)

where the content of client.csr is provided as public key string ...

The CSR is not a public key, it is a CSR. But the code you show clearly doesn't use it. It does apparently use the certificate, which also is not a public key, it is a certficate -- which contains a public key. However, a certificate, and NOT a public key OR a CSR, is the correct thing to use for SSL/TLS authentication.

I get PKCS#1 private key as String input and I have to use bouncy castle to convert to PKCS#8 and generate privatekey and add to store ....

You don't actually need to convert to PKCS8; PEMParser plus JcaPEMKeyConvertor can convert PKCS1 directly to a KeyPair containing a usable (RSA)PrivateKey, which is much simpler. Or OpenSSL can easily create PKCS8 key which standard JCE (without BouncyCastle) can read -- or for that matter PKCS12 which is already a Java keystore without any conversion of any kind. But what you have does work.

Now I have to create java keystore, add these keys and export as JKS file pragmatically.

Aside: you probably mean programmatically, not pragmatically. Your code doesn't export any keystore file, and doesn't need to; and you don't need to use a JKS keystore specifically, any supported in-memory keystore should work, although you don't show the code doing the connection and using the keystore. Starting with Java 9 (almost two years ago) Oracle recommends people stop using JKS and use PKCS12 instead, although I doubt they will actually drop JKS anytime soon.

javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2

That's your actual problem, and it's not the keystore, or key, and shouldn't be the cert. Even if you are sending a key and cert it doesn't like, the server should respond by rejecting the authentication within the protocol(s), not violating protocol. And 2 is ServerHello, which should occur only before the certificate is sent or checked in either direction. Do at least one of the following:

• standard Java SSL/TLS debugging: run with sysprop javax.net.debug=ssl,handshake and capture/log the output. (Below Java 11 you can actually use just ssl and it automatically includes handshake.) This will show exactly what your system is sending and receiving, and allow determining how it is wrong, which should at least help determine why.

• capture the handshake with a network trace like wireshark, tcpdump, or similar, and look at it. This works even if there is something seriously wrong in Java (which there shouldn't be) but is at least a little harder to decode, and if you aren't quite careful risks including other data on the network during the same time period that may be sensitive and unshareable.

• try to connect with another tool (or tools), using the same key+cert and from the same machine if at all possible -- this might be a network-related problem that depends on where you connect from. Since you have openssl and already have the files in OpenSSL formats, simplest is to use

  openssl s_client -connect $host:$port -key keyfile -cert certfile [see text]

  s_client is also unusual (unique AFAIK) in that you can control whether or not SNI (the Server Name Indication extension) is sent, and with many servers nowadays that may well affect the server's behavior (possibly triggering or suppressing a bug). Whether Java (JSSE) sends SNI depends on a combination of usually several factors you did not tell us: always the version of Java, and usually some details of the code making the connection attempt, especially whether it uses HttpsURLConnection, the new-j11+ HttpClient, third-party code like Apache or google, or custom code. For OpenSSL releases through 1.1.0, SNI is not sent by default and you must add -servername $host to send it. For 1.1.1, it is sent by default but is not sent if you add -noservername.

  In this example I did not specify a truststore. s_client by default will verify against its default truststore, which may or may be correct for your server, but will ignore any error and proceed anyway. If you were debugging a certificate trust problem then it would be important to match OpenSSL's truststore with the one used by your Java, but not for your problem.

• check with the server operator(s) how this problem is perceived at their end: do they think there is something wrong with your handshake, and if so what? What software (or usually more relevant middleware) are they running, with what config or options?

If you add (some/enough of) this information to your Q, I will try to update this to be an actual answer.