SSL Socket Connection: CFNetwork SSLHandshake failed (-9824) connecting to Java SSLServerSocket

Question

I am setting up a secure socket connection using the SSLServerSocket class in Java and then the Stream classes (both CF and NS) in objective c. I have it set up so with the following code:

Java SSLServerSocket code (SecureClientWorker deals with each connection):

public void listenSocket(int port){
        try {
            System.setProperty("javax.net.ssl.keyStore", "path/to/certificate(signed_by_own_root_certificate).jks");
//I try to make this root certificate trusted by iOS (see obj-c code below)
            System.setProperty("javax.net.ssl.keyStorePassword", "PASSWORD...");
        } catch (IOException e1) {
            e1.printStackTrace();
        }
        try{
            SSLServerSocketFactory ssf = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
            server = (SSLServerSocket) ssf.createServerSocket(port);

            logger.info("Started to listen on port: "+port);
        }catch(Exception e){
            logger.warning("Could not listen on port: "+port);
            e.printStackTrace();
        }

        while(running){
            SecureClientWorker w;
            try{
                w = new SecureClientWorker(server.accept(), this);
                Thread t = new Thread(w);
                t.start();
            }catch(IOException e){
                if(running)
                    e.printStackTrace();
            }
        }

    }

Objective C client code:

[self addRootCert]; //This is where I attempt to make the root certificate trusted, If needed I can post it
CFReadStreamRef readStream;
CFWriteStreamRef writeStream;
CFStreamCreatePairWithSocketToHost(NULL, (__bridge CFStringRef)self.ip.text, self.portNumber.text.intValue, &readStream, &writeStream);

//    Set options then bridge to ARC:
NSDictionary *settings = [[NSDictionary alloc] initWithObjectsAndKeys:
                          (id)kCFStreamSocketSecurityLevelNegotiatedSSL, kCFStreamPropertySocketSecurityLevel,
                          [NSNumber numberWithBool:YES], kCFStreamSSLAllowsExpiredCertificates,
                          [NSNumber numberWithBool:YES], kCFStreamSSLAllowsAnyRoot,
                          [NSNumber numberWithBool:YES], kCFStreamSSLAllowsExpiredRoots,
                          [NSNumber numberWithBool:NO], kCFStreamSSLValidatesCertificateChain,
                          nil];

if (readStream) {
    CFReadStreamSetProperty(readStream, kCFStreamPropertySSLSettings, (__bridge CFDictionaryRef)settings);
    inStream = (__bridge_transfer NSInputStream*) readStream;
    [inStream setDelegate:self];
    [inStream scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
    [inStream open];
}

if (writeStream) {
    CFWriteStreamSetProperty(writeStream, kCFStreamPropertySSLSettings, (__bridge CFDictionaryRef)settings);
    outStream = (__bridge_transfer NSOutputStream*) writeStream;
    [outStream setDelegate:self];
    [outStream scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
    [outStream open];
}

However this results in the error code CFNetwork SSLHandshake failed (-9824) and the inStream object returns the error NSStreamEventErrorOccurred in the NSStreamDelegate method.

On the Java side, an IOException appears to get thrown with the following:

in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream())); //ClientSocket is just the connection to the client
[...]
in.readLine();

The printStackTrace says : javax.net.ssl.SSLHandshakeException: no cipher suites in common

Do I have to create a KeyManager(as I read somewhere else)? And how do I do this (from nothing to having an opened SSLServerSocket please)?

Any help would be greatly appreciated!

EDIT

I have changed the server code since it may not be able to set system properties.

My new code is:

KeyStore ks = KeyStore.getInstance("JKS");
            FileInputStream ksIs = new FileInputStream(new File("path/to/certificate(signed_by_own_root_certificate).jks"));
            try {
                ks.load(ksIs, "PASSWORD...".toCharArray());
            } catch (NoSuchAlgorithmException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            } catch (CertificateException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            } finally {
                if (ksIs != null) {
                    ksIs.close();
                }
            }

            KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
            kmf.init(ks, "PASWD".toCharArray());
            KeyManager keyManagers[] = kmf.getKeyManagers();
            SSLContext sc = SSLContext.getDefault();
            sc.init(keyManagers, null, new SecureRandom());
            SSLServerSocketFactory socketFactory = sc.getServerSocketFactory();

            server = (SSLServerSocket) socketFactory.createServerSocket(port);

However I now get this error message: java.security.KeyManagementException: Default SSLContext is initialized automatically

After some research I tried to make my own TrustManager but couldn't get it to work still

What am I missing?

Answer 1

Assuming that the keystore file can be read by your application and that the password is correct, my guess is that you've made use of SSL/TLS connection in your Java application somewhere before getting this socket factory (this might be done implicitly by another library).

The javax.net.ssl.keyStore* system properties are only used once to initialise the default SSLContext. Changing them after that context has been initialised will do nothing.

The javax.net.ssl.SSLHandshakeException: no cipher suites in common error message is quite typical of a keystore not being found or loaded properly. This comes from the fact that, without any certificate/key material, none of the RSA/DSA cipher suites are enabled: these are the ones the remote client will be looking for.

A quick way to check this would be to set these properties on the command line, or to set them at the very beginning of your application (and not to change them elsewhere on your server code).

If you need these settings not to be global in your application, you'll need to initialise a KeyManager indeed.

---

Following your edit, java.security.KeyManagementException: Default SSLContext is initialized automatically happens because you're trying to re-configure the default SSLContext. Use a new instance instead:

SSLContext sc = SSLContext.getInstance("TLS");