CODEWITHSUNDEEP
pythonsslpython-requestsopenssl
psytester
psytester

Reputation: 244

Root cause found in openSSL due to AKI DirName extension: Python [SSL: CERTIFICATE_VERIFY_FAILED] ...: unable to get local issuer certificate

in short

I got: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

what is a much worse error than [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

And I suspected python or my own code, but the same error occurs with pure openSSL:

openssl s_client -connect my-domain.com:443 -CAfile root.pem -verify 2

   ......

    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
......

If openSSL does not work, python will not work, too. Fore sure this is for any error at TLS handshake. ;-)

Comparing old and new certificates, the root cause was finally found in two new addtional Authority Key Identifier entries DirName and serial. They where not present before:

openssl x509 -in my-domain.pem -text

The only diff and root cause was Authority Key Identifier with two new entries:

X509v3 Authority Key Identifier:
    keyid:1D:E8:38:95:85:65:A2:D9:44:99:96:30:D1:81:D5:5B:F7:38:CC:8C
    DirName:/C=DE/O=My Company/OU=My OU/CN=My Sub-CA
    serial:02

New and final question: Why openSSL doesn't work with DirName and serial within AKI?

I keep all the initial stuff, because that might help other to build up flexible python code.....

Now, all my approaches below are working fine, as expected.

If CA ist not in truststore I get the expected python error:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)
Setting truststore at context, session or requests level, take what you want, it works for me.

situation last 2 years

In past (2 years ago) I imported my private CA (root and sub-ca) into Win 10 local computer truststore
and used this code snippet to create my own urllib3 context to load the Windows store with context.load_default_certs()

That worked until this week, where I had to renew my CA and server certificate. :-(
I deleted old private CA bundle and imported new CA into Windows local computer truststore and firefox truststore as usual.
All browser are working fine with new truststore!

new Problem after update of my private CA

With python requests I now get again these annoying errors:

SSLError in steps code - Message: HTTPSConnectionPool(host='my-domain.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

my approaches in code

My current code tries different ways to solve the issue, but nothing works.
I'm sure (or hope) the fault is at my side, but I don't see it anymore ;-)

import certifi
import json
import os
import requests

    class MyHttpAdapter(HTTPAdapter):
        def init_poolmanager(self, *args, **kwargs):
            # Google: "SSL failure on Windows using python requests" -- usage of Windows Truststore
            # https://stackoverflow.com/questions/42981429/ssl-failure-on-windows-using-python-requests
            context = create_urllib3_context()

I tried those 3 settings to load different ca bundles (only one code line is active per attempt)

            ## --------------------
            # variant 1: usage of Windows Truststore where I imported my root-ca and sub-ca into local computer storage
            # load_default_certs() --> this loads the OS defaults of Windows Truststore!!!!
            context.load_default_certs()
            
            # variant 2: my special ca bundle contains only my private CA only with 3 entries: root-ca, sub-ca and server cert
            # load_verify_locations() --> this loads just a specific CA bundle !!!!
            context.load_verify_locations(cafile="C:/Users/...../my_ca_bundle.pem")
            
            # variant 3: I added my private CA with root-ca, sub-ca and server cert to certifi cacert.pem file as a hack
            # certifi.where(): C:\Users\........\Python39\Lib\site-packages\certifi\cacert.pem
            # load_verify_locations() --> this loads just a specific CA bundle !!!!
            context.load_verify_locations(cafile=certifi.where())
            ## --------------------

To verify that my private CA is within the selected store, I do an assert on the CommonName of my CA, thus it would fail fast if not found. If it's fine, the context is added to the session.mount later on:

            # get and print all CAs from loaded list to verify that my CA is within the list
            # notice the server certificate of pem file is not loaded into the list, don't no why, maybe because it's no CA
            json_ca_certs = context.get_ca_certs()
            print("All current context CA certs: {}".format(json.dumps(json_ca_certs, indent=1, sort_keys=True)))
            length=len(json_ca_certs)
            print("Number of certificates in bundle: {}".format(length))

            # fail fast:
            assert "MY CommonName" in str(json_ca_certs), "private CA not found in used bundle"

            kwargs['ssl_context'] = context
            return super().init_poolmanager(*args, **kwargs)

I do not set those environment variables, they are None

    def __init__(self, **kwargs):

        print("REQUESTS_CA_BUNDLE: '{}'".format(os.environ.get('REQUESTS_CA_BUNDLE')))
        print("CURL_CA_BUNDLE: '{}'".format(os.environ.get('CURL_CA_BUNDLE')))

I create a session and as creation of context with my truststore does not longer work, I try 3 different ways to use session.verify (one code line active per attempt only):

        self.session = requests.Session()

        # variant 1: default not working anymore:
        self.session.verify = True

        # variant 2: usage of my special ca bundle does not work
        self.session.verify = "C:/Users/...../my_ca_bundle.pem"

        # variant 3: certifi cacert.pem file contains my private CA and does not work
        # certifi.where(): C:\Users\........\Python39\Lib\site-packages\certifi\cacert.pem
        self.session.verify = certifi.where()

Here I finally mount my adapter to the session:

        # mount with Schema only as prefix to use them for all my calls (this worked in past and should not be a new problem)
        adapter = self.MyHttpAdapter()
        self.session.mount("https://", adapter)

Now I do my self.session.get() calls. Once again I tried those settings, since context cafile nor session.verify works, direct at requests within session:

    _api_url = 'https://my-domain.com/api/function'
    
    ###

    # variant 1: does not longer work, no setting above for session.verify nor context.load...():
    response = self.session.get(_api_url)

    # variant 2: verify = my special ca bundle work around does not work:
    response = self.session.get(_api_url, verify="C:/Users/...../my_ca_bundle.pem")
    
    # variant 3: verify = certifi cacert.pem work around does not work:
    response = self.session.get(_api_url, verify=certifi.where())

I added to requests/sessions.py some print debug output, to see what is used
sessions request() used final settings:

        settings = self.merge_environment_settings(
            prep.url, proxies, stream, verify, cert
        )
        print(f"DEBUG: request settings: {settings}")

sessions merge_environment_settings() right before the return ()

        print(f"DEBUG: merge_environment_settings() before return: verify={verify}")
        return {'verify': verify, 'proxies': proxies, 'stream': stream,
                'cert': cert}

Just to see, that my settings are used:

DEBUG: merge_environment_settings() before return: verify=path_to_ca_bundle.pem
DEBUG: request settings: {'verify': 'path_to_ca_bundle.pem', 'proxies': OrderedDict(), 'stream': False, 'cert': None}

Python and installed modules are up-to-date:
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21)
certifi==2021.10.8
requests==2.27.1
requests-toolbelt==0.9.1

Upvotes: 2

Views: 825

Answers (1)

psytester
psytester

Reputation: 244

The initial entry with short question and solution is already much too long. As it works now with new CA and server certificate, I will put latest question to this answer.

In rfc3280 4.2.1.1 Authority Key Identifier

The authority key identifier extension provides a means of
identifying the public key corresponding to the private key used to
sign a certificate.  This extension is used where an issuer has
multiple signing keys (either due to multiple concurrent key pairs or
due to changeover).  The identification MAY be based on either the
key identifier (the subject key identifier in the issuer's
certificate) or on the issuer name and serial number.

The keyIdentifier field of the authorityKeyIdentifier extension MUST
be included in all certificates generated by conforming CAs to
facilitate certification path construction......

...
id-ce-authorityKeyIdentifier OBJECT IDENTIFIER ::=  { id-ce 35 }

AuthorityKeyIdentifier ::= SEQUENCE {
  keyIdentifier             [0] KeyIdentifier           OPTIONAL,
  authorityCertIssuer       [1] GeneralNames            OPTIONAL,
  authorityCertSerialNumber [2] CertificateSerialNumber OPTIONAL  }

Notice: All three AuthorityKeyIdentifier attributes are OPTIONAL and not linked together.
But in the paragraph before one could read:
The identification MAY be based on either the 1. key identifier (the subject key identifier in the issuer's certificate) or on the 2. issuer name and serial number.

In contrast we see the NOTE in rfc4158 3.5.12 "Matching Key Identifiers (KIDs)"

...
NOTE:  Although required to be present by [RFC3280], it is extremely
important that KIDs be used only as sorting criteria or as hints
during certification path building.  KIDs are not required to match
during certification path validation and cannot be used to eliminate
certificates.  This is of critical importance for interoperating
across domains and multi-vendor implementations where the KIDs may
not be calculated in the same fashion.

It's a sorting criteria only and not a validation criteria.

And finally in openSSL FAQ 15 "Why does OpenSSL set the authority key identifier (AKID) extension incorrectly?"

It doesn’t: this extension is often the cause of confusion.
Consider a certificate chain A->B->C so that A signs B and B signs C. Suppose certificate C contains AKID.
The purpose of this extension is to identify the authority certificate B. This can be done either by including the subject key identifier of B or its issuer name and serial number.
In this latter case because it is identifying certificate B it must contain the issuer name and serial number of B.
It is often wrongly assumed that it should contain the subject name of B. If it did this would be redundant information because it would duplicate the issuer name of C.

So, they mean this???
This can be done either by including

  1. the subject key identifier of B
    --> or <--
  2. its issuer name and serial number.
    In this latter case because it is identifying certificate B it must contain the issuer name and serial number of B.

I'm finally confused for this week...

Upvotes: 1

Related Questions