Authenticating a SSL server against a pre-obtained copy of its self-signed certificate

Question

I'm trying to communicate between a C# client and server with an SslStream. However my server's certificate is self-signed.

The client owns a copy of the public part of the server's certificate, assumed to be transmitted over a tamper-proof (but not listen-proof) channel.

Can I securely use this to authenticate the server, and to do so, what should I put in the client's SslStream or its RemoteCertificateValidationCallback? So far all my attempts using validation have failed due to lack of trust, and what my callback does for now is compare the names and fingerprints.

If acceptable, how can I get the client to "trust" its copy of the public certificate for validation?

Answer 1

The first solution is simple but still has vulnerabilities. That is to just check that the key fingerprint matches. But, any middle man can establish an intermediary connection and create his own key exchange and then craft yours to match his, sharing encrypting keys at that point.

The fundamental problem is that you don't have a proper infrastructure. It's not that you're using self signed certificates per se, but it's that you're using self signed certificates from the same "CA". You need to create a root certificate for your certification authority (yourself) and then from that key you need to approve a CSR for your piece of software to use. This will mean that all remote machines need to trust your root certificate and install it as a certificate authority in their machine. By doing this you would avoid the issue of having the certificate used by the application being compromised mid-stream or anything like that. But this only applies if you can get that root CA certificate out to the users in a guaranteed tamper-proof fashion.

A different but far more drastic solution to your problem is to essentially ditch SSL or do out of band authentication for that certificate's validity. The client can verify the certificate because the server is able to decrypt the client's messages and make appropriate responses. A challenge response scheme like this can mathematically/cryptographically prove that the server has the expected certificate or at least that the party has the corresponding private key for it:

Client generates a 256 bit random nonce (RNGCryptoServiceProvider), K_C
Server generates a 256 bit random nonce, K_S
Client encrypts K_C with server's assumed public key, now K_B
Server digitally signs K_S, K_A
Server transmits K_A to client
Client transmits K_B to server
Server decrypts K_B, now N_S
Client verifies K_A using server's assumed public key, if invalid disconnect
Client computes K_A XOR K_C, now Z_C
Server computes N_S XOR K_A, now Z_S
Server transmits Z_S to client
Client verifies Z_S matches Z_C
Client transmits Z_C to server, encrypted using the server's public key
Server decrypts Z_C and verifies that it matches Z_S

If everything checks out, the challenge is complete and you can guarantee the server is in possession of the private key for the public key you have. You also can't just shortcut this and transmit a digitally signed copy of the server's public key to the client becaue then any middle man can create two sessions on either side This scheme prevents two sessions from being established by a man in the middle because the client is in possession of the public key already and challenges the validity directly by encrypting their nonce with the server's public key. The best attack on this is a man in the middle observer that can't tamper with the data without exploiting a weakness in the asymmetric algorithm being used.

Another benefit to this scheme is that you can just avoid SSL all together at this point, since you have authenticated independently. Now, you can actually use Z_S == Z_C to your advantage and consider Z_S or Z_C (they're identical) as your agreed upon symmetric key for further communication.

I wrote a paper on a scheme similar to this a while ago and it was published a few places. You can probably find it by the name of "Challenging Authentication-Agreement Protocol (CAAP)", dated 2007 or revised in 2010-ish under my name. There are code examples.