Windows 2019, Python 3.9.7, urllib, certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

Question

Python scripts on Windows 2019 have started to fail for me, apparently due to botched local truststore and I seem to be unable to make it right.

Stacktrace

C:\tmp\DEL (master -> origin)
λ python
Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request;file_name, headers = urllib.request.urlretrieve("https://repo1.maven.org/fromsearch?filepath=org/antlr/antlr4-runtime/4.9.2/antlr4-runtime-4.9.2.jar");println(file_name);
Traceback (most recent call last):
  File "C:\Python39\lib\urllib\request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\Python39\lib\http\client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Python39\lib\http\client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Python39\lib\http\client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Python39\lib\http\client.py", line 1034, in _send_output
    self.send(msg)
  File "C:\Python39\lib\http\client.py", line 974, in send
    self.connect()
  File "C:\Python39\lib\http\client.py", line 1448, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "C:\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python39\lib\urllib\request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "C:\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python39\lib\urllib\request.py", line 517, in open
    response = self._open(req, data)
  File "C:\Python39\lib\urllib\request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Python39\lib\urllib\request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\Python39\lib\urllib\request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>
>>>

Context

I tried updating Python certs, e.g.

    Start-Process -Wait -FilePath "C:\Python39\python.exe" -ArgumentList "-m","pip","install","--upgrade","pip"
    Start-Process -Wait -FilePath "C:\Python39\python.exe" -ArgumentList "-m","pip","install","python-certifi-win32"
    Start-Process -Wait -FilePath "C:\Python39\python.exe" -ArgumentList "-m","pip","install","--upgrade","certifi"

I also tried downloading the certificate chain and doing:

set REQUESTS_CA_BUNDLE=C:\tmp\repo1-maven-org-chain.pem

And last but not least, out of sheer desperation, I also did try (although I do not want that as a "solution"):

set PYTHONHTTPSVERIFY=0

For more context, I update the systems cert store during installation thus:

function Import-509Certificate([String]$certPath,[String]$certRootStore,[String]$certStore,[String]$myAlias) {
    # Windows keystore
    $pfx = new-object System.Security.Cryptography.X509Certificates.X509Certificate2
    $pfx.import($certPath)
    $store = new-object System.Security.Cryptography.X509Certificates.X509Store($certStore,$certRootStore)
    $store.open("MaxAllowed")
    $store.add($pfx)
    $store.close()

    # Java store
    $javaStores = @(
        "C:\Program Files\openjdk-11\lib\security\cacerts"
        "C:\Program Files\openjdk-8\jre\lib\security\cacerts"
    )
    foreach($store in $javaStores) {
        Start-Process -Wait -FilePath "C:\Program Files\openjdk-8\bin\keytool" -ArgumentList "-noprompt","-importcert","-alias","`"$myAlias`"","-keystore","`"$store`"","-storepass","changeit","-file","`"$certPath`""
    }
}

function Install-Certs() {
    $letsEncryptCerts = @(
    "https://letsencrypt.org/certs/isrgrootx1.pem"
    "https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem"
    "https://letsencrypt.org/certs/lets-encrypt-r3.pem"
    "https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem"
    "https://letsencrypt.org/certs/lets-encrypt-e1.pem"
    )
    foreach($url in $letsEncryptCerts) {
        $certFile = $(Split-Path -Path $url -Leaf)
        Write-Host "Downloading $url to C:\tmp\$certFile"
        $wc.DownloadFile($url, "C:\tmp\$certFile")
        Import-509Certificate "C:\tmp\$certFile" "LocalMachine" "Root" $certFile
    }
    Import-509Certificate "C:\cygwin64\tmp\legacy.pem" "LocalMachine" "Root" "rh_it_legacy"
    Import-509Certificate "C:\cygwin64\tmp\maven-org-chain.pem" "LocalMachine" "Root" "maven_org_chain"
}

I am not much of either Python or Windows user, but I understand that Python runtime itself has its own trust store, independent on the system one.

Is there any clean solution that might help? I am not looking for switching off the trust chain verification. I am looking for a way to update truststore to make the Python runtime trust the signature.

Latest updated Firefox installed on that Windows system trusts that site (https://repo1.maven.org) without any warning.

Answer 1

Problem solved. Modern Python such as the version used in the question, i.e. 3.7.9 is actually using Windows own trust store.

Downloading correct GlobalSign certificates and storing them in the trust store as "TrustedPublisher" solved the problem:

"https://secure.globalsign.net/cacert/Root-R1.crt"
"https://secure.globalsign.net/cacert/Root-R3.crt"

and

$store = New-Object System.Security.Cryptography.X509Certificates.X509Store "TrustedPublisher","LocalMachine"
$store.Open("ReadWrite")
$store.Add($certPath)
$store.Close()

Python happily trusts the site now:

>>> import urllib.request;file_name, headers = urllib.request.urlretrieve("https://repo1.maven.org/fromsearch?filepath=org/antlr/antlr4-runtime/4.9.2/antlr4-runtime-4.9.2.jar");print(file_name);

C:\Users\ADMINI~1\AppData\Local\Temp\2\tmpvc1zkphg