CODEWITHSUNDEEP
kubernetesgoogle-kubernetes-enginegitlab-ci-runner
Christopher
Christopher

Reputation: 421

Losing my mind trying to get the VM to talk to Kubernetes in Google Cloud

I've been pulling my hair out over this for too many hours... I'm pretty new to kubernetes so I know I must be missing something.

"ERROR: Job failed (system failure): the server does not allow access to the requested resource (post pods)"

We have a GitLab instance setup on a VM, and another VM with the GitLab runner installed. Both live in Google Cloud Compute Engine.

We also have a Kubernetes cluster spun up on Google Cloud.

When the runner attempts to run, it results in the following:

Running with gitlab-runner 10.0.2 (a9a76a50)
  on rd-002-optic-nexus (21590677)
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image docker:git ...
ERROR: Job failed (system failure): the server does not allow access to the requested resource (post pods)

Due to the Runner being "external" to the cluster, my only option is to authenticate to the API server via "client certificate" authentication.

I'm using the cluster ca.crt provided from the Google Cloud Console, and have followed Kubernetes guide to create a client cert. However, I just can't get it to work.

I must be missing something somewhere.

GitLab Runner Config

concurrent = 1
check_interval = 0
[[runners]]
  name = "rd-002-optic-nexus"
  url = "https://our.gitlab.instance.com/"
  token = "21590677f31b57bce610ef3f4cb20d"
  executor = "kubernetes"
  [runners.kubernetes]
    host = "https://111.222.x.xxx"
    cert_file = "/usr/local/share/ca-certificates/kube-client.crt"
    key_file = "/usr/local/share/ca-certificates/kube-client.key"
    ca_file = "/usr/local/share/ca-certificates/kubernetes-ca.crt"
    namespace = "gitlab"
    namespace_overwrite_allowed = ""
    privileged = false
    cpu_limit = "1"
    memory_limit = "1Gi"
    service_cpu_limit = "1"
    service_memory_limit = "1Gi"
    helper_cpu_limit = "500m"
    helper_memory_limit = "100Mi"
    [runners.kubernetes.node_selector]
      gitlab = "true"

Kubernetes Client CSR

apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: gitlab-sa.gitlab
spec:
  groups:
  - system:authenticated
  request: $(cat server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth

Any thoughts? Anything I'm missing?

Upvotes: 2

Views: 226

Answers (2)

silverfox
silverfox

Reputation: 5272

The user account provided must have permission to create, list and attach to Pods in the specified namespace in order to function.

-- Connecting to the Kubernetes API

You need create a role and binding to the user gitlab-sa.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: gitlab
  name: gitlab-runner-executor
rules:
- apiGroups: [""]
  resources: ["pods", "pods/attach"]
  verbs: ["create", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-runner
  namespace: gitlab
subjects:
- kind: User
  name: gitlab-sa
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: gitlab-runner-executor
  apiGroup: rbac.authorization.k8s.io

Normally the ServiceAccount gitlab-sa is used by Gitlab Runner within the Kubernetes cluster.

It's maybe not necessary when running Gitlab Runner externally to the Cluster, I'm not very sure.

Upvotes: 0

Robert Bailey
Robert Bailey

Reputation: 18230

Your client certificate has a usage for server auth instead of client auth. For your TLS client to use the certificate to authenticate to the Kubernetes apiserver it needs to provide a certificate with the client auth usage.

Upvotes: 1

Related Questions