How to debug ERR_SSL_PROTOCOL_ERROR / SSL_ERROR_RX_RECORD_TOO_LONG?

Question

I've written simple Java HTTPS server.

Creating server socket:

KeyStore keyStore = KeyStore.getInstance("JKS");
keyStore.load(
    getCertificateKeyStore().getIterator().toInputStream(),
    getCertificateKeyStorePassword().toCharArray()
);

KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("SunX509");
keyManagerFactory.init(keyStore, getCertificateKeyPassword().toCharArray());
KeyManager[] km = keyManagerFactory.getKeyManagers();

TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance("SunX509");
trustManagerFactory.init(keyStore);
TrustManager[] tm = trustManagerFactory.getTrustManagers();

SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(km, tm, null);
SSLServerSocketFactory sslServerSocketFactory = sslContext.getServerSocketFactory();
return sslServerSocketFactory.createServerSocket(port);

Generating certificate:

final SystemConsole systemConsole = new SystemConsole();
systemConsole.writeLine("apt install -y certbot");
systemConsole.writeLine(
    "certbot certonly " + (
        "--standalone " +
        "--non-interactive " +
        "--agree-tos " +
        "--email " + email + " " +
        "--domains " + new JoinStrings(domainNames, ' ')
    )
);
System.out.println(systemConsole.read());
final Directory storage = getCommonDataStorage();
final Directory letsEncryptLiveDir = Directory.getExisting("/etc/letsencrypt/live");
final Directory letsEncryptKeysDir = letsEncryptLiveDir.getNotNull(letsEncryptLiveDir.getKeys().getFirst());
storage.removeIfExists("certificate.jks");
systemConsole.writeLine("cd " + letsEncryptKeysDir.absolutePath);
systemConsole.writeLine("openssl pkcs12 -export -out keystore.pkcs12 -in fullchain.pem -inkey privkey.pem -password pass:ilovejava");
System.out.println(systemConsole.read());
systemConsole.write("keytool -importkeystore"); {
    systemConsole.write(" -srckeystore keystore.pkcs12");
    systemConsole.write(" -srcstoretype PKCS12");
    systemConsole.write(" -destkeystore " + storage.absolutePath + "/certificate.jks");
    systemConsole.write(" -storepass " + keyStorePassword);
    systemConsole.write(" -keypass " + keyPassword);
    systemConsole.write(" -srcstorepass ilovejava");
    systemConsole.writeLine();
}
System.out.println(systemConsole.read());
systemConsole.writeLine("rm keystore.pkcs12");
return storage.read("certificate.jks");

And it used to work well until yesterday. Then it SUDDENLY stopped working.

In Chrome: ERR_SSL_PROTOCOL_ERROR

In Firefox: SSL_ERROR_RX_RECORD_TOO_LONG

In Edge: strangely still works.

Another strange thing: Chrome say my SSL certificate is actually OK:

Moreover! Chrome also establish connection, send request and receive response (at least its headers):

And it looks like it does it over SECURE socket connection (because there are no other way to connect). And handshake is actually occur (as I can see in server logs).

But still, I get that SSL related errors. How can I debug what EXACTLY goes wrong?

UPDATE: Tried openssl s_client -connect <my_domain>:443. Getting this:

CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = stage.bd-polpharma.com.ua
verify return:1
---
Certificate chain
 0 s:/CN=stage.bd-polpharma.com.ua
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFaTCCBFGgAwIBAgISBB3XbH/2SQ6Ai1WXqKyLIDJjMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTA0MjUwNzQ3NDlaFw0x
OTA3MjQwNzQ3NDlaMCQxIjAgBgNVBAMTGXN0YWdlLmJkLXBvbHBoYXJtYS5jb20u
dWEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDF0iM/tA0qu644/RUv
m3uGA6RjbPihNnXQW2qy2RAALJhCprXDoVpMGzxIrZs/XvXvcRTa7ps9+zPJCCRN
dUxnydm0rk5H5Orvt2gWMCT/Ln42OyHHjDuYFo3kY9U8xHrkY+x2r4uHBv1GDkQ/
HZrmHRk6fy0yeaMKx6j8y2H8BvG3Hwj7p73sW2umUxnuoP4dbRSTBZqTk6dGNjuN
lWonCFandADcIa2fwzVkBUUDo4NegxB1SuVsu7mGegzmyxCLQhIo8NtSDeCuEN6N
MV8YVenP3CMsHpHAbx1Ly8FbLpmIOUTmDAE4OkNQJTmHEJ2s+ZVT8WNlfhh+rUSf
fUqrAgMBAAGjggJtMIICaTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFItV0gKguULL
OGG03Q4KJXJnL58uMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8G
CCsGAQUFBwEBBGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxl
dHNlbmNyeXB0Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxl
dHNlbmNyeXB0Lm9yZy8wJAYDVR0RBB0wG4IZc3RhZ2UuYmQtcG9scGhhcm1hLmNv
bS51YTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsG
AQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQMGCisGAQQB1nkC
BAIEgfQEgfEA7wB2AG9Tdqwx8DEZ2JkApFEV/3cVHBHZAsEAKQaNsgiaN9kTAAAB
alOt8a4AAAQDAEcwRQIhAPRmn7ySWOg2bwY90b1q6HxEYyv/APvH2R6a+lwljXJ1
AiAHxeuBUoveypoA5kUueihWe8HjdOuIF0JDbqvCks2itwB1ACk8UZZUyDlluqpQ
/FgH1Ldvv1h6KXLcpMMM9OVFR/R4AAABalOt8OEAAAQDAEYwRAIgT6LbDXae36hY
WFCNA7uybp95ta2ql0iVq0spxttOawECIBbu9Z/oBRAlrGhiMshL9Gqq4ok8OciR
6NmOsH3nXqtlMA0GCSqGSIb3DQEBCwUAA4IBAQAJYXoBWnhdLITZxL0I5s/IrerC
ychitXGqxXY8yJd+HxikJnz+tw6QU+De/aZFdu7/zyZID0X/f59bSE6C2EJZXccP
NQB0Qi7XcMUddMLiHT5vXhQ+KIqctbQyUNokxz8blB6rEi2zDhoHgfvFHQP4ErWG
pJYXLDTMYMT1kdF4ciyiauKLMxPc6i1Mn0Ob0CYLtBl9suLEBrw5RS9ZSlVNXwFN
rl6Vrrvs6H5ntMffBC4zqHELhwTx/uFoh50XzYH7LeLKoGdl5UwRLiPMAJ/zPnDy
BqdhRghbHcTXbbM1qiuO90juZzazUh6rOtK4hDZ97jraacMC1EiaaLdTCYSE
-----END CERTIFICATE-----
subject=/CN=stage.bd-polpharma.com.ua
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3069 bytes and written 302 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: AC9F77EB2B148C44D053707331C431BE1EA40944B6FB2C67FF3B7F4D9B8B00F8
    Session-ID-ctx: 
    Master-Key: AC4B43FE48F138613B55D5E42E39998A5A9500E86B7B154D177D3E98B3DC2A03D5CC344052DE2310B8C1CE91BD487AA5
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1556186839
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

UPDATE: Tried getDefaultAlgorithm(), "PKIX", "TLSv1.2", "TLSv1.3" with all possible combinations. Didn't help.

Answer 1

After 3 days of trying every possible thing and asking every possible person, I found that I can downgrade Java version from 11 to 10 (OpenJDK or Oracle JDK doesn't matter). And it solves the problem.

But still, why? And how to deal with it on later Java? By the way, Oracle JDK 12 still has this... bug? I'm not sure how to call it. It can't be that SSL is broken in Java 11+. I think, I'm doing something wrong, but I still don't know what.