CODEWITHSUNDEEP
dockerkubernetesgitlabgitlab-cigitlab-ci-runner
andreas.teich
andreas.teich

Reputation: 911

Not able to execute GitLab Runner in Kubernetes cluster: cannot create resource "secrets" in API group "" in the namespace "gitlab"

Currently I'm facing the issue:

ERROR: Job failed (system failure): 
prepare environment: 
setting up credentials: 
secrets is forbidden: 
User "system:serviceaccount:default:gitlab-runner" cannot create
resource "secrets" in API group "" in the namespace "gitlab"

after following the official documentation on how to integrate the GitLab Runner.

I'm using the following runner-chart-values.yaml:

# The GitLab Server URL (with protocol) that want to register the runner against
# ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
#
gitlabUrl: http://example.domain/

# The Registration Token for adding new runners to the GitLab Server. This must
# be retrieved from your GitLab instance.
# ref: https://docs.gitlab.com/ce/ci/runners/README.html
#
runnerRegistrationToken: "<token>"

# For RBAC support:
rbac:
    create: true
    rules:
      - apiGroups: ["*"]

# Run all containers with the privileged flag enabled
# This will allow the docker:dind image to run if you need to run Docker
# commands. Please read the docs before turning this on:
# ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind
runners:
    privileged: true

Any clues what's going on?

Many thanks!

Upvotes: 14

Views: 21788

Answers (7)

Camilla
Camilla

Reputation: 1

You have to enable RBAC support. Or disable RBAC on your Kubernetes installation.

There is a section in the official documentation addressing this error: https://docs.gitlab.com/runner/install/kubernetes.html#error-job-failed-system-failure-secrets-is-forbidden.

Upvotes: 0

Syakur Rahman
Syakur Rahman

Reputation: 2102

In addition to the other answers. Here is a link to the official documentation which lists what permissions are needed depending on the strategy used:

enter image description here

Link to documentation: https://docs.gitlab.com/runner/executors/kubernetes.html

Upvotes: 0

Sachitha Hirushan
Sachitha Hirushan

Reputation: 643

I also got the same error. So I have used this method. It solved my error.

helm install --namespace <NAMESPACE> gitlab-runner -f values.yml --set rbac.create=true gitlab/gitlab-runner

Upvotes: 2

cglacet
cglacet

Reputation: 10932

Here is a complete solution using Helm, I copied the rights proposed by Richard in this answer.

Using the followin template (gitlab-rbac/templates) we could patch a given namespace using:

helm upgrade -i gitlab-rbac-name ./gitlab-rbac \
-n your-namespace-here --create-namespace

Once installed, you can check your current rights with:

kubectl auth can-i create secrets --as=system:serviceaccount:gitlab:default \
-n your-namespace-here

The template gitlab-rbac/templates/rbac.yaml contains the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ .Release.Namespace }}-admin
  namespace: {{ .Release.Namespace }}
rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["list", "get", "watch", "create", "delete"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get"]
    - apiGroups: [""]
      resources: ["pods/attach"]
      verbs: ["list", "get", "create", "delete", "update"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["list", "get", "create", "delete", "update"]      
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["list", "get", "create", "delete", "update"]  
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gitlab-runner-{{ .Release.Namespace }}-admin
  namespace: {{ .Release.Namespace }}
subjects:
  - kind: ServiceAccount
    name: default
    namespace: gitlab
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: {{ .Release.Namespace }}-admin

Note that you might need a lot more rights for your runners, you will probably need to update the rules depending on your pipeline. For example if you allow your template to create namespaces, you will need to add a cluster wide role for this. This means adding the following in the template file:

# ... Role and RoleBinding templates
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: namespace-admin
rules:
  - apiGroups: ['']
    resources: ['namespaces']
    verbs: ['create']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gitlab-runner-namespace-admin
subjects:
  - kind: ServiceAccount
    name: default
    namespace: gitlab
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: namespace-admin

The value (gitlab-rbac/values.yaml) file is empty in this example as we only use the namespace argument.

Upvotes: 4

Richard
Richard

Reputation: 161

For me adding all necessary roles was the only solution that actually helped.

Here the corresponding runner-chart-values.yaml file:

## GitLab Runner Image
gitlabUrl: http://example.domain/
runnerRegistrationToken: "<token>"

rbac:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["list", "get", "watch", "create", "delete"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get"]
    - apiGroups: [""]
      resources: ["pods/attach"]
      verbs: ["list", "get", "create", "delete", "update"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["list", "get", "create", "delete", "update"]      
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["list", "get", "create", "delete", "update"]      

runners:
  privileged: true

Upvotes: 16

Bazhikov
Bazhikov

Reputation: 841

Extending Harsh's answer: Please make sure that you're working under active 'gitlab-runner' namespace or using the key --namespace=gitlab-runner. To switch between active namespaces, please use the following command:

kubens gitlab-runner

So you don't need to use --namespace=gitlab-runner everytime.

JFYI, I've done that steps from the article on my k8s cluster and it works fine for me.

Upvotes: 0

Harsh Manvar
Harsh Manvar

Reputation: 30110

Looks like there is namespace mismatch however you can try this below option

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "get", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]

make sure you are creating the service account of Role to proper namespace.

Command to create Role binding

kubectl create rolebinding --namespace=gitlab-runner gitlab-runner-binding --role=gitlab-runner --serviceaccount=gitlab-runner:default

here is nice documentation : https://medium.com/@ruben.laguna/installing-a-gitlab-runner-on-kubernetes-ac386c924bc8

Upvotes: 5

Related Questions