DH ServerKeyExchange does not comply to algorithm constraints

Question

We recently upgraded one of our tools (uses java implementation) and it now has a hard time connecting to one of our company internal endpoints. For other endpoints it works fine. Lets say the endpoint against which it doesn't work is xyz.abc.com The error we see each time is

javax.net.ssl.SSLHandshakeException: DH ServerKeyExchange does not comply to algorithm constraints
        at sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
        at sun.security.ssl.DHServerKeyExchange$DHServerKeyExchangeConsumer.consume(DHServerKeyExchange.java:540)
        at sun.security.ssl.ServerKeyExchange$ServerKeyExchangeConsumer.consume(ServerKeyExchange.java:111)
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
        at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
        at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961)
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:933)

I followed the suggestions under :-

What does "Reason: DHPublicKey does not comply to algorithm constraints" mean?

However that didn't seem to work.

Since this issue is prevalent now only after our upgrade of the tool and we wonder if it could be due to newer jvm security restrictions being enforced,is there anything we could possibly pass as jvm argument or may be just modify under the java.security file or elsewhere to fix this annoying error.We do not want to change anything on the certificates used by that endpoint. I am a novice in cryptography and hence any help or suggestions here would be greatly appreciated as always.

Answer 1

According to the man page crypto-policies(7), we should create a custom subpolicy definition, which lowers the minimum bits for DH key exchange (min_dh_size) to 1024 for the java-tls scope.

So we have to created /etc/crypto-policies/policies/modules/LEGACY-DH-JAVA.pmod as

min_dh_size@java-tls = 1024

And updated the system wide policy

$ update-crypto-policies --show
DEFAULT

$ sudo update-crypto-policies --set DEFAULT:LEGACY-DH-JAVA
Setting system policy to DEFAULT:LEGACY-DH-JAVA
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.

Note: System-wide crypto policies are applied on application start-up.

So we have to restart our Java application after this change.

Answer 2

On a Red Hat Enterprise Linux 8.5 system I resolved this by setting the system into LEGACY mode like this. More info here https://access.redhat.com/articles/3666211

$ update-crypto-policies --show
DEFAULT

$ sudo update-crypto-policies --set LEGACY
Setting system policy to LEGACY
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.

$ sudo reboot

Answer 3

Just like to share for those who also encounter the same issue and have no luck changing one value.

There's a good chance that

• the certificate of your endpoint is using a DH keysize 1024.
• in java.security, jdk.tls.disabledAlgorithms is set to disable DH keySize < 2048

In that case, original solution which contains "jdk.tls.disabledAlgorithms=DH keySize < 1024" worked.

In Linux like environment, you could verify the DH keysize used to sign the certificate of your host.

openssl s_client -connect <host>:<port> -msg

You should find this line from above result

Server Temp Key: DH, 1024 bits

For those who find updating java.security file alone did not do the trick, chances are there's an external file that overrides some settings. Look for the below setting in java.security. Either have this config set to false, or update the external file as original post did.

security.overridePropertiesFile=true

While Oracle discourages the use of weaker cryptographic algorithms, it is up to you whether/when to follow suite.

Oracle JRE and JDK Cryptographic Roadmap

Answer 4

I was able to fix this by modifying the java crypto-policies as :-

jdk.tls.ephemeralDHKeySize=1024
jdk.certpath.disabledAlgorithms=MD2, MD5, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=DH keySize < 1024, SSLv2, SSLv3, TLSv1, TLSv1.1, DHE_DSS, RSA_EXPORT, DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_DSS_EXPORT, DH_RSA_EXPORT, DH_anon, ECDH_anon, DH_RSA, DH_DSS, ECDH, 3DES_EDE_CBC, DES_CBC, RC4_40, RC4_128, DES40_CBC, RC2, HmacMD5
jdk.tls.legacyAlgorithms=

I had to replace 2048 with 1024 for DHKeySize