CODEWITHSUNDEEP
pythonkubernetesgoogle-kubernetes-enginekubernetes-helm
markwalker_
markwalker_

Reputation: 12859

Mounting a certificate for using the GKE Kubernetes API

I'm porting a django management command to a new [private] GKE cluster configured with service accounts & workload identity. This command uses the kubernetes API to change settings on the autoscaler for the cluster.

It looks like the API connection requires a token and a certificate. These are bundled up to create the configuration;

    configuration = kubernetes.client.Configuration()
    configuration.api_key["authorization"] = token
    configuration.api_key_prefix["authorization"] = "Bearer"
    configuration.host = server
    configuration.ssl_ca_cert = cert

    api = kubernetes.client.AutoscalingV1Api(
        kubernetes.client.ApiClient(configuration)
    )

The existing project that I'm porting this command from uses defaults for token and certificate which are defined as;

    parser.add_argument(
        "--cert",
        action="store",
        dest="cert",
        default="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
        help="File containing valid certificate to make request",
    )
    parser.add_argument(
        "--token",
        action="store",
        dest="token",
        type=argparse.FileType("r"),
        default="/var/run/secrets/kubernetes.io/serviceaccount/token",
        help="File containing token to make request",
    )

I've noticed that these aren't added by GKE by default. And looking at the pods for the existing project, I can see that /var/run/secrets doesn't exist. So I think this cluster is able to use this API via it's default service account, whereas this new cluster doesn't use that SA.

The error I'm seeing come from attempts to run this command point at the missing certificate;

HTTPSConnectionPool(host='10.255.240.1', port=443): Max retries exceeded with url: /apis/autoscaling/v1/namespaces/staging/horizontalpodautoscalers/draft-nginx (Caused by SSLError(FileNotFoundError(2, 'No such file or directory')))

I found the google docs on how I can mount a token. So the helm for that is in my templates and I've verified the token in a pod;

          containers:
            - name: scale-workloads
              image: {{ .Values.gke_registry }}/base_python:{{ .Values.global.build }}
              imagePullPolicy: Always
              command:
                - python -m django
              args:
                - scale_workloads
                - --namespace={{ .Release.Namespace }}
                - --appserver={{ .Values.pods.appserver.minReplicas | default 1 }}
                - --nginx={{ .Values.pods.nginx.minReplicas | default 1 }}
              env:
                {{- include "proj.sharedEnv" $ | nindent 16 }}
                - name: DJANGO_SETTINGS_MODULE
                  value: {{ .Values.django_settings_module }}
              resources:
                requests:
                  cpu: 1000m
                  memory: 500Mi
              volumeMounts:
                - mountPath: /etc/config
                  name: configs
                - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
                  name: ksa-token
          volumes:
          - name: configs
            projected:
              defaultMode: 420
              sources:
              - secret:
                  name: proj-secrets
          - name: ksa-token
            projected:
              sources:
                - serviceAccountToken:
                    path: ksa-token
                    expirationSeconds: 86400
                    audience: some-oidc-audience

But can't find any similar docs on mounting a certificate that the cluster either is, or could, be using.

The stacktrace from manually running this management command shows the following;

File "/usr/src/app/drafty/core/management/commands/scale_workloads.py", line 198, in scale_pods
    api.patch_namespaced_horizontal_pod_autoscaler(
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/autoscaling_v1_api.py", line 983, in patch_namespaced_horizontal_pod_autoscaler
    return self.patch_namespaced_horizontal_pod_autoscaler_with_http_info(name, namespace, body, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/autoscaling_v1_api.py", line 1098, in patch_namespaced_horizontal_pod_autoscaler_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 407, in request
    return self.rest_client.PATCH(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 296, in PATCH
    return self.request("PATCH", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 169, in request
    r = self.pool_manager.request(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.255.240.1', port=443): Max retries exceeded with url: /apis/autoscaling/v1/namespaces/staging/horizontalpodautoscalers/draft-nginx (Caused by SSLError(FileNotFoundError(2, 'No such file or directory')))

Upvotes: 1

Views: 118

Answers (1)

markwalker_
markwalker_

Reputation: 12859

I've solved this by getting the certificate from the cluster;

import base64

import kubernetes.client
from google.auth import compute_engine
from google.cloud.container_v1 import ClusterManagerClient
from kubernetes.client.rest import ApiException
from python_hosts import Hosts, HostsEntry

...

        # Get cluster details
        credentials = compute_engine.Credentials()

        cluster_manager_client = ClusterManagerClient(credentials=credentials)
        cluster = cluster_manager_client.get_cluster(
            name=f"projects/{settings.GCP_PROJECT}/locations/{settings.GCP_REGION}/clusters/{settings.GCP_PROJECT}"
        )

        # Save cluster certificate for SSL verification
        cert = base64.b64decode(cluster.master_auth.cluster_ca_certificate)
        cert_filename = "cluster_ca_cert"
        with open(cert_filename, "wb") as cert_file:
            cert_file.write(cert)

        # Configure hostname for SSL verification
        hosts = Hosts()
        hosts.add(
            [
                HostsEntry(
                    entry_type="ipv4",
                    address=cluster.endpoint,
                    names=["kubernetes"],
                )
            ]
        )
        hosts.write()

        configuration = kubernetes.client.Configuration()
        configuration.host = f"https://{cluster.endpoint}:443"
        configuration.api_key["authorization"] = token
        configuration.api_key_prefix["authorization"] = "Bearer"
        configuration.ssl_ca_cert = cert_filename

        kubernetes.client.Configuration.set_default(configuration)

        api = kubernetes.client.AutoscalingV1Api(
            kubernetes.client.ApiClient(configuration)
        )

Upvotes: 0

Related Questions