CODEWITHSUNDEEP
javassl
egorlitvinenko
egorlitvinenko

Reputation: 2776

SSL handshake exception: "Algorithm constraints check failed: MD5withRSA"

I tried to install Oracle Entitlements Server Client. When I call 

config.cmd -smConfigId Sample-SM -prpFileName C:\oracle\product\11.1.2\as_1\oessm\SMConfigTool\smconfig.java.controlled.prp

I got this Exception:

    javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
        at java.io.OutputStream.write(OutputStream.java:75)
        at oracle.security.oes.enroll.EnrollmentClient.writeToSocket(EnrollmentClient.java:330)
        at oracle.security.oes.enroll.EnrollmentClient.enroll(EnrollmentClient.java:161)
        at oracle.security.oes.enroll.EnrollmentClient.main(EnrollmentClient.java:478)
        at oracle.security.oes.tools.EnrollmentTool.doEnroll(EnrollmentTool.java:103)
        at oracle.security.oes.tools.SMConfigTool.doEnrollment(SMConfigTool.java:1192)
        at oracle.security.oes.tools.SMConfigTool.run(SMConfigTool.java:617)
        at oracle.security.oes.tools.SMConfigTool.main(SMConfigTool.java:546)
    Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:350)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
        ... 15 more
    Caused by: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:159)
        at sun.security.provider.certpath.PKIXCertPathValidator.doValidate(PKIXCertPathValidator.java:351)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:191)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:279)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:345)
        ... 21 more
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA

Can you help me to find a reason?

Upvotes: 18

Views: 60235

Answers (2)

Igor Nardin
Igor Nardin

Reputation: 1631

The problem is caused by Oracle disabling hash algorithms which are no longer considered to be secure. Take a look at

JRE_HOME/lib/security/java.security

It contains the following properties:

jdk.certpath.disabledAlgorithms
jdk.tls.disabledAlgorithms

You can adjust them appropriately. For example, remove MD5 from the former and MD5withRSA from the latter.

Hint for docker images:

there is additional config file /etc/crypto-policies/back-ends/java.config in some docker images like keycloak in my case which overrides values in java.security

Upvotes: 52

egorlitvinenko
egorlitvinenko

Reputation: 2776

keyser gave direction for answer in comment.

Problem was in key's length. In short: "Starting from 7u40, the use of x.509 certificates with RSA keys less than 1024 bits in length is restricted."

So the right way to solve this problem it is using certificates with key's length at least 2048 bits.

Upvotes: 5

Related Questions