SSL Client-Server with self signed certificate imported at runtime fails on handshake

Question

EDIT: I fixed the code. See my answer for explanation.

I am trying to create a android app that sends data to a server using a SSLSocket. The certificate is self signed and I am trying add it in the server KeyManager and client TrustManager using SSLContext, as per instructions I found on other threads.

However both the client and server return exceptions when I try to send data.

Please keep in mind that this is my first time dealing with SSL connections. I created and singed the 'server.crt' certificate using the Java keytool.

client exception

javax.net.ssl.SSLHandshakeException: Handshake failed
at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:390)
at com.android.org.conscrypt.OpenSSLSocketImpl.waitForHandshake(OpenSSLSocketImpl.java:623)
at com.android.org.conscrypt.OpenSSLSocketImpl.getOutputStream(OpenSSLSocketImpl.java:609)
...
Caused by: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0x73c6ac00: Failure in SSL library, usually a protocol error

server exception

Exception in thread "main" javax.net.ssl.SSLHandshakeException: no cipher suites in common
...
at java.io.InputStream.read(InputStream.java:101)

FIXED client code (running on android)

AssetManager assetManager = getAssets();
InputStream keyStoreInputStream = assetManager.open("keystore.bks");

KeyStore keyStore = KeyStore.getInstance("BKS");
keyStore.load(null,null);
keyStore.load(keyStoreInputStream, KEY_PASSWORD);

TrustManagerFactory trustManagerFactory =
    TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagerFactory.getTrustManagers(), null);

SSLSocketFactory sslsocketfactory = sslContext.getSocketFactory();
SSLSocket sslsocket = (SSLSocket) sslsocketfactory.createSocket("192.168.1.212", 4444);

InputStream inputStream = assetManager.open(MainActivity.FILE_NAME);

OutputStream outputStream = sslsocket.getOutputStream();

int count;
byte[] buffer = new byte[12 * 1024];
while ((count = inputStream.read(buffer)) > 0) {
    outputStream.write(buffer, 0, count);
}

FIXED server code (running on windows)

FileInputStream keyStoreInputStream = new FileInputStream(KEYSTORE_PATH);

KeyStore keyStore = KeyStore.getInstance("JKS");
keyStore.load(null, null);
keyStore.load(keyStoreInputStream, KEY_PASSWORD);

KeyManagerFactory keyManagerFactory =
    KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagerFactory.init(keyStore, KEY_PASSWORD);

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagerFactory.getKeyManagers(), null, null);

SSLServerSocketFactory sslServerSocketFactory = sslContext.getServerSocketFactory();
SSLServerSocket sslServerSocket =
    (SSLServerSocket) sslServerSocketFactory.createServerSocket(4444);

while (true) {
    SSLSocket sslSocket = (SSLSocket) sslServerSocket.accept();

    InputStream inputStream = sslSocket.getInputStream();

    FileOutputStream outputStream = new FileOutputStream(OUTPUT_PATH,false);

    byte[] bytes = new byte[12 * 1024];
    int count;
    while ((count = inputStream.read(bytes)) > 0) {
        outputStream.write(bytes, 0, count);
    }
}

Answer 1

For testing purposes, I needed to spin up an HTTPS server. As the cert didn't really matter, I needed to create one. Why leave the JVM and use keytool when we can do it all in java.

This requires BouncyCastle.

static SSLContext createSSLContext() throws Exception {
    final KeyStore ks = createSelfSignedKeyStore();
    final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
    kmf.init(ks, new char[0]);

    final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
    tmf.init(ks);

    final SSLContext sslContext = SSLContext.getInstance("TLS");
    sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);
    return sslContext;
}

private static KeyStore createSelfSignedKeyStore() throws Exception {
    final KeyStore ks = KeyStore.getInstance("JKS");
    ks.load(null);

    final KeyPairGenerator keyGen = KeyPairGenerator.getInstance("RSA");
    keyGen.initialize(1024);
    final KeyPair keyPair = keyGen.generateKeyPair();
    final Certificate selfSignedCert = generate(InetAddress.getLocalHost().getCanonicalHostName(), keyPair);

    ks.setCertificateEntry("alias.cert", selfSignedCert);
    ks.setKeyEntry("alias.key", keyPair.getPrivate(), new char[0], new Certificate[] { selfSignedCert });
    return ks;
}

private static Certificate generate(final String fqdn, final KeyPair keypair) throws Exception {
    final Instant now = Instant.now();
    final Date notBefore = Date.from(now.minus(24, ChronoUnit.HOURS));
    final Date notAfter = Date.from(now.plus(24, ChronoUnit.HOURS));
    final ContentSigner selfSigner = new JcaContentSignerBuilder("SHA256WithRSAEncryption")
        .build(keypair.getPrivate());
    final X500Name owner = new X500Name("CN=" + fqdn);
    final X509CertificateHolder certHolder = new JcaX509v3CertificateBuilder(
        /*issuer*/owner,
        /*serial*/new BigInteger(64, new SecureRandom()),
        /*cert start*/notBefore,
        /*cert end*/notAfter,
        /*subject*/owner,
        /*subject public key*/keypair.getPublic())
        .build(selfSigner);
    final X509Certificate selfSignedCert = new JcaX509CertificateConverter()
        .getCertificate(certHolder);
    selfSignedCert.verify(keypair.getPublic()); // Certificate is good!
    return selfSignedCert;
}

Answer 2

It seems that the problem was with the certificate I was using. I tried to import the '*.crt' file created with the keytool and then add it to a keystore programmatically. Even though I am still not sure why it didn't work before, I fixed this by skipping the certificate import and imported the keystore directly ('keystore.jks' on windows server and 'keystore.bks' on android client).

This is how I created the keystore.jks:

keytool -genkeypair -alias serveralias -keyalg RSA -keysize 4096 -keystore keystore.jks 
        -keypass changeit -storepass changeit -validity 10950

Since JKS is not supported on android I made a copy of the keystore and renamed the extension to .bks on the new one. Then I used a program called 'KeyStore Explorer' to convert the keystore.bks to 'BKS-V1' format so it works on android.

I updated the original question to include the working code.

Answer 3

I think the key piece of information is the server-side exception: Exception in thread "main" javax.net.ssl.SSLHandshakeException: no cipher suites in common. Your self-signed certificate may be fine; it looks like you have a mismatch in the cipher configurations. You need to make sure that both the client and the server have a cipher suite in common. The default supported suites for the SSLSocket in Android depend upon the version. See the release notes for Android 5.0