Unable to `openssl verify' letsencrypt certificate

Question

I gererate a certificate with Letsencrypt using the Certbot container:

$ mkdir /home/$USER/letsencrypt
$ docker run -it --rm -p 80:80 -p 443:443 -v /home/$USER/letsencrypt:/etc/letsencrypt certbot/certbot certonly --standalone --email user@example.com --agree-tos -d example.com

I navigate to the generated certificate:

$ cd /home/$USER/letsencrypt/live/example.com

I can verify chain.pem:

$ openssl verify chain.pem 
chain.pem: OK

And I can see what's in chain.pem:

$ openssl x509 -noout -in chain.pem -subject -issuer
subject=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

I can't verify cert.pem (presumably because it needs the chain):

$ openssl verify cert.pem
CN = example.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error cert.pem: verification failed

But I also can't verify fullchain.pem either:

$ openssl verify fullchain.pem
CN = example.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error fullchain.pem: verification failed

The certificate seems to work in the browser, but is failing in curl (and an Android http client, which is the real issue):

$ curl https://example.com
curl: (60) SSL certificate problem: unable to get local issuer certificate

I've double-checked that fullchain.pem is a concatenation of cert.pem and chain.pem.

So: I don't understand why fullchain.pem doesn't verify?

Answer 1

The problem is the last cert in a certbot issued fullchain is a cert that is issued by the expired root cert (output in json to illustrate each cert in the chain) as of Sept 30, 2021 (https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/)

[
  {
    "subject": {
      "commonName": "..."
    },
    "issuer": {
      "countryName": "US",
      "organizationName": "Let's Encrypt",
      "commonName": "R3"
    },
    "version": 3,
    ...
    },
    "OCSP": [
      "http://r3.o.lencr.org"
    ],
    "caIssuers": [
      "http://r3.i.lencr.org/"
    ]
  },
  {
    "subject": {
      "countryName": "US",
      "organizationName": "Let's Encrypt",
      "commonName": "R3"
    },
    "issuer": {
      "countryName": "US",
      "organizationName": "Internet Security Research Group",
      "commonName": "ISRG Root X1"
    },
    "version": 3,
    "serialNumber": "912B084ACF0C18A753F6D62E25A75F5A",
    "notBefore": "Sep  4 00:00:00 2020 GMT",
    "notAfter": "Sep 15 16:00:00 2025 GMT",
    "caIssuers": [
      "http://x1.i.lencr.org/"
    ],
    "crlDistributionPoints": [
      "http://x1.c.lencr.org/"
    ]
  },
  {
    "subject": {
      "countryName": "US",
      "organizationName": "Internet Security Research Group",
      "commonName": "ISRG Root X1"
    },
    "issuer": {
      "organizationName": "Digital Signature Trust Co.",
      "commonName": "DST Root CA X3"
    },
    "version": 3,
    "serialNumber": "4001772137D4E942B8EE76AA3C640AB7",
    "notBefore": "Jan 20 19:14:03 2021 GMT",
    "notAfter": "Sep 30 18:14:03 2024 GMT",
    "caIssuers": [
      "http://apps.identrust.com/roots/dstrootcax3.p7c"
    ],
    "crlDistributionPoints": [
      "http://crl.identrust.com/DSTROOTCAX3CRL.crl"
    ]
  }
]

Note that the last item in the chain is issued by DST Root CA X3, and if you fetch http://apps.identrust.com/roots/dstrootcax3.p7c you'll see it is the newly expired DST Root CA X3 cert.

So you get

$ openssl verify -CAfile fullchain.pem fullchain.pem
server.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 1 depth lookup:certificate has expired
O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup:certificate has expired
OK

But if you strip the last cert from fullchain.pem and put the output in chain.pem

$ openssl verify -CAfile chain.pem fullchain.pem
server.pem: OK

(note that verify only checks the first cert in fullchain.pem)

With recent versions of openssl you can use -partial_chain or -trusted_first but those are unavailable on the openssl installed on MacOS.

I have a few pem processing tools here: https://gitlab.com/Blockdaemon/pem2json

Answer 2

I was struggling with the same issue for 3 days. But the error was a result of a configuration error in the in my Apache configuration.

I found out by Command openssl s_client -connect advertentiekracht.nl:443 returned:

Certificate chain
 0 s:/CN=advertentiekracht.nl
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3

inclusive the "Unable to get local issuer certificate"

Command : [root@srv ssl]# openssl x509 -noout -in /etc/letsencrypt/live/advertentiekracht.nl/chain.pem -subject -issuer showed the missing chain:

subject= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3

I am certainly not familiar with openssl and certificates. There certainly can be a lot of reasons leading to "Unable to get local issuer certificate. But before you start digging like I did, check your http server configuration. For me that is Apache. I had typos in the where the SSL certificate hocus pocus is defined. The httpd toke the erroneous lines below

        SSLCertificateFile /etc/letsencrypt/live/advertentiekracht.nl/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/advertentiekracht.nl/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

Mind the first line SSLCertificateFile shoud be SSLCertificateChainFile, and I missed the references to the cert.pem and the chain.pem. The lines below solved my problem:

        SSLCertificateChainFile /etc/letsencrypt/live/advertentiekracht.nl/fullchain.pem
    SSLCertificateFile /etc/letsencrypt/live/advertentiekracht.nl/cert.pem
    SSLCertificateChainFile /etc/letsencrypt/live/advertentiekracht.nl/chain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/advertentiekracht.nl/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

Result, a complete chain:

    Certificate chain
 0 s:/CN=advertentiekracht.nl
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Answer 3

I figured this out from man verify, reading the description of untrusted. Turns out untrusted is actually how you specify the certificate chain of trust (seems counterintuitive when you put it like that).

So, the command you need to verify a Letsencrypt cert is:

openssl verify -untrusted chain.pem cert.pem

Where cert.pem is your certificate and chain.pem is the LE intermediate cert. There's no need to use fullchain.pem for this.

Answer 4

Counterintuitively, I finally got openssl verify to work by adding the root certificate to the chain. It feels like the Letsencrypt CA should already be available, so I'm not convinced this is the right thing to do (and would welcome comments).

The steps were:

• Chrome developer tools > Security tab > View Certificate > Details tab > Select root certificate ("Builtin Object Token:DST Root CA X3")
• Click Export, export as Base64-Encoded ASCII, Single certificate (I named it ca.pem)

Concatenate the root to the chain:

$ ca.pem fullchain.pem > cachain.pem

Then verify:

$ openssl verify cachain.pem
cachain.pem: OK

This feels "wrong" so I'd like to understand whether this is a false positive.