TLS Handshake error after server hello Done

Question

I have a python app is connecting to a java app, both use the same certificate. This certificate is also imported in trusted-certs of both sides.

The functionality of the whole TLS handshake process was validated with a self-signed certificate.

However when I want to use a globally CA-signed certificate instead, encounter a handshake error:

This is the SSL log I got by enabling -Djavax.net.debug=ssl,handshake on the server-side (java app):

.
.
.
***
update handshake state: certificate[11]
upcoming handshake states: server_key_exchange[12](optional)
upcoming handshake states: certificate_request[13](optional)
upcoming handshake states: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
*** ECDH ServerKeyExchange
Signature Algorithm SHA256withRSA
Server key: Sun EC public key, 256 bits
  public x coord: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  public y coord: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  parameters: secp256r1 [NIST P-256, X9.62 prime256v1] (1.2.840.10045.3.1.7)
update handshake state: server_key_exchange[12]
upcoming handshake states: certificate_request[13](optional)
upcoming handshake states: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
*** ServerHelloDone
update handshake state: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
Thread-54, WRITE: TLSv1.2 Handshake, length = 2075
Thread-54, READ: TLSv1.2 Alert, length = 2
Thread-54, RECV TLSv1.2 ALERT:  fatal, unknown_ca
%% Invalidated:  [Session-2, SSL_NULL_WITH_NULL_NULL]
%% Invalidated:  [Session-4, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
Thread-54, called closeSocket()
Thread-54, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca

The last lines, aftrer ServerHelloDone, indicate the error:

Thread-54, WRITE: TLSv1.2 Handshake, length = 2075
Thread-54, READ: TLSv1.2 Alert, length = 2
Thread-54, RECV TLSv1.2 ALERT:  fatal, unknown_ca
%% Invalidated:  [Session-2, SSL_NULL_WITH_NULL_NULL]
%% Invalidated:  [Session-4, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
Thread-54, called closeSocket()
Thread-54, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca

The next step is apparently client_key_exchange. Is it related to the client's private key in any way?

What does unknown_ca mean? Is the certificate missing in the client's trust_cert or in server's?

The below image is also the traffic that was captured using Wireshark, primary lines are using the self-signed certificate, and the last 3 lines are for the troubled certificate, discussed above:

Answer 1

What does unknown_ca mean? Is the certificate missing in the client's trust_cert or in server's?

unknown_ca means that the certificate is not trusted due to a missing CA in the trust store. Since the alert is sent by the client it means that the client does not trust the CA which issued the servers certificate. This might be because the root CA is missing in the trust store but it might also be that the server did not send the necessary intermediate certificates so that the client could build the trust chain to the local root CA.