Python kubernetes client requests fail with "unable to get local issuer certificate"

Question

I'm using the supported python Kubernetes library (pip install kubernetes, v12.0.1). When I make a request to my cluster for any option (eg to create a Namespace), it fails with the following error:

import kubernetes.client

def create_namespace(self, namespace_key):                            
    with kubernetes.client.ApiClient(self.configuration) as client:
        api = kubernetes.client.CoreV1Api(client)                  
        api.create_namespace(                                      
            {                                                      
                "apiVersion": "v1",                                
                "kind": "Namespace",                               
                "metadata": {                                      
                    "name": namespace_key,                        
                },                                                 
            }                                                      
        )                                                          


urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<url of my cluster>', port=443): Max retries exceeded with url: /api/v1/namespaces (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer
 certificate (_ssl.c:1123)')))

If I turn off SSL configuration using self.configuration.verify_ssl = False in the client, the request works, but it gives me a warning telling me that disabling SSL is strongly discouraged.

My setup:

• Ubuntu 20.04.1 LTS (running in a VM on mac OSX)
• Python 3.8.5 (virtual env, no pyenv or pipenv)
• The virtualenv has certifiy==2020.11.8 installed (not sure if this impacts SSL interactions or not)

Is there anything system-wide that I need to do locally and/or to the docker container that will eventually be running these requests in production?

EDIT A bit more info:

>>> import ssl                
>>> print(ssl.OPENSSL_VERSION)
OpenSSL 1.1.1f  31 Mar 2020   

Answer 1

This ended up being my mistake: my hosted Kubernetes provider (Digital Ocean) provides a custom TLS certificate with Kubernetes credentials, and that needs to be used as the certificate for the k8s python client.

In general, you might see this regardless of hosting provider if the certificate used in your client doesn't match what the k8s API is expecting.

Specifically, this was how I resolved the issue for Digital Ocean:

    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {DIGITAL_OCEAN_ACCESS_TOKEN}",
    }
    url = (
        f"https://api.digitalocean.com/v2/kubernetes/clusters/{cluster_id}/credentials"
    )
    response = requests.get(url, headers=headers)
    data = json.loads(response.content.decode("utf-8"))
    if response.status_code != 200:
        raise Exception(f"Unable to authenticate to cluster {cluster_id}")
    my_clusters_host_url = data["server"]
    my_new_api_key = data["token"]

    # Decode and save certificate
    decoded_certificate = base64.b64decode(data["certificate_authority_data"])
    certificate_authority_data = str(decoded_certificate, "utf-8")

As far as I know, the client doesn't let you set the cert data directly; it needs to be saved to a file. So I create a tmp file below:

        filepath = f"/tmp/cluster_{cluster.id}.crt"
        with open(filepath, "w+") as f:
            f.write(certificate_authority_data)
        configuration = kubernetes.client.Configuration(
            host=host_url,
            api_key={
                "authorization": f"Bearer {api_key}",
            },
        )
        configuration.ssl_ca_cert = filepath

And then that configuration object can be used in other API requests. For example:

        with kubernetes.client.ApiClient(configuration) as client:
            api = kubernetes.client.CoreV1Api(client)
            api.create_namespace(payload)