Is there a way to make Azure Application Gateway return all certificates in the chain?

Question

We're migrating a web app from AWS to Azure. Some users access the site through an app on embedded tablets that run Android 6. I'm getting certificates from LetsEncrypt using certbot and, inside a renew hook, converting the certificate to pfx and updating the application gateway:

openssl pkcs12 -inkey $RENEWED_LINEAGE/privkey.pem -in $RENEWED_LINEAGE/cert.pem -certfile $RENEWED_LINEAGE/chain.pem -export -out $RENEWED_LINEAGE/cert.pfx -passout pass:$PASSWORD
az network application-gateway ssl-cert update -g $RGNAME --gateway-name $AGNAME -n $DOMAIN --cert-file $RENEWED_LINEAGE/cert.pfx --cert-password $PASSWORD

If I inspect this, I can see that it contains three certificates:

openssl pkcs12 -in $RENEWED_LINEAGE/cert.pfx -passin pass:$PASSWORD -passout pass:$PASSWORD

subject=CN = new.(redacted).com
issuer=C = US, O = Let's Encrypt, CN = R3
...
subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
...
subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

However, once the cert has been applied to the application gateway, when I connect to it the chain appears to omit the TrustID X3 Root (DST Root CA X3) certificate:

openssl s_client -debug -connect new.(redacted).com:443

Certificate chain
0 s:CN = new.(redacted).com
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 28 (HH:mm:ss) 2023 GMT; NotAfter: Oct 26 (HH:mm:ss) 2023 GMT
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT

This works fine for modern clients - Chrome reports the connection is secure - but Android 6 clients fail with the exception "javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found."

I have tried specifically downloading the ISRG Root X1 cert (cross signed by TrustID X3 Root (DST Root CA X3)) and the self-signed TrustID X3 Root cert, and including them in the pfx:

curl -s -o $RENEWED_LINEAGE/isrgrootx1.pem "https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem"
curl -s -o $RENEWED_LINEAGE/trustidx3.pem "https://letsencrypt.org/certs/trustid-x3-root.pem.txt"
cat $RENEWED_LINEAGE/chain.pem $RENEWED_LINEAGE/isrgrootx1.pem $RENEWED_LINEAGE/trustidx3.pem > $RENEWED_LINEAGE/certs.pem
openssl pkcs12 -inkey $RENEWED_LINEAGE/privkey.pem -in $RENEWED_LINEAGE/cert.pem -certfile $RENEWED_LINEAGE/certs.pem -export -out $RENEWED_LINEAGE/cert.pfx -passout pass:$PASSWORD

I could confirm that there were more certificates in the pfx, but after applying it to the application gateway there aren't any more certificates in the returned chain. I also tried using fullchain.pem instead of chain.pem when converting to pfx, but this fails with a duplicate certificate error.

I'm confident that the application gateway not returning some certificates in the chain is the issue, because if I connect to the existing site (in AWS) it returns a chain with four certificates, including the missing Trust ID X3 root:

openssl s_client -debug -connect old.(redacted).com:443

Certificate chain
0 s:CN = old.(redacted).com
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 15 01:mm:ss 2023 GMT; NotAfter: Oct 13 01:mm:ss 2023 GMT
1 s:CN = old.(redacted).com
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 15 01:mm:ss 2023 GMT; NotAfter: Oct 13 01:mm:ss 2023 GMT
2 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
3 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT

The Android 6 tablets have no issues connecting to the existing site. The fact that the first certificate gets returned twice (the subject CN and timestamps are identical) make me think that someone has specifically fiddled with the certificate bundle, possibly to fix this issue.

Answer 1

This is a known issue with Azure Application Gateway v1. It correctly uses the full certificate bundle in Application Gateway v2.

As far as I can tell this isn't documented anywhere, but Azure Support are aware of it. Application Gateways perform a validation process when given a certificate bundle, and it appears that if it finds a chain to what it considers a trusted root it ignores any further certificates and uses a (partial) chain.

To encounter this issue you need:

• a certificate signed by a CA which is trusted by Azure,
• but isn't a trusted root for your client,
• and the CA cert is cross-signed by a different CA,
• which is trusted by the client.

I ran into this using LetsEncrypt and a pre-7.1.1 Android client, but it's probably uncommon and can generally be resolved by either using Application Gateway v2, using a different CA, or dropping support for older Android clients.