I am getting permission issue (cannot create resource \"Job\" in API group \"batch) while creating jobs via sensors of argo-events

Question

I am trying to trigger a job creation from a sensor but I am getting the error below:

   Job.batch is forbidden: User \"system:serviceaccount:samplens:sample-sa\" cannot create resource \"Job\" in API group \"batch\" in the namespace \"samplens\"","errorVerbose":"timed out waiting for the condition: Job.batch is forbidden: User \"system:serviceaccount:samplens:sample-sa\" cannot create resource \"Job\" in API group \"batch\" in the namespace \"samplens\"\nfailed to execute trigger\ngithub.com/argoproj/argo-events/sensors.(*SensorContext).triggerOne\n\t/home/jenkins/agent/workspace/argo-events_master/sensors/listener.go:328\ngithub.com/argoproj/argo-events/sensors.(*SensorContext).triggerActions\n\t/home/jenkins/agent/workspace/argo-events_master/sensors/listener.go:269\ngithub.com/argoproj/argo-events/sensors.(*SensorContext).listenEvents.func1.3\n\t/home/jenkins/agent/workspace/argo-events_master/sensors/listener.go:181\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","triggerName":"sample-job","triggeredBy":["payload"],"triggeredByEvents":["38333939613965312d376132372d343262302d393032662d663731393035613130303130"],"stacktrace":"github.com/argoproj/argo-events/sensors.(*SensorContext).triggerActions\n\t/home/jenkins/agent/workspace/argo-events_master/sensors/listener.go:271\ngithub.com/argoproj/argo-events/sensors.(*SensorContext).listenEvents.func1.3\n\t/home/jenkins/agent/workspace/argo-events_master/sensors/listener.go:181"}
12

Although I have created a serviceaccount, role and rolebinding. Here is my serviceaccount creation file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sample-sa
  namespace: samplens

Here is my rbac.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sample-role
  namespace: samplens
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - create
      - delete
      - get
      - watch
      - patch
  - apiGroups:
      - "batch"
    resources:
      - jobs
    verbs:
      - create
      - delete
      - get
      - watch
      - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sample-role-binding
  namespace: samplens
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: sample-role
subjects:
  - kind: ServiceAccount
    name: sample-sa
    namespace: samplens

and here is my sensor.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: webhook
spec:
  template:
    serviceAccountName: sample-sa
  dependencies:
    - name: payload
      eventSourceName: webhook
      eventName: devops-toolkit
  triggers:
    - template:
        name: sample-job
        k8s:
          group: batch
          version: v1
          resource: Job
          operation: create
          source:
            resource:
              apiVersion: batch/v1
              kind: Job
              metadata:
                name: samplejob-crypto
                annotations:
                  argocd.argoproj.io/hook: PreSync
                  argocd.argoproj.io/hook-delete-policy: HookSucceeded
              spec:
                ttlSecondsAfterFinished: 100
                serviceAccountName: sample-sa
                template:
                  spec:
                    serviceAccountName: sample-sa
                    restartPolicy: OnFailure
                    containers:
                      - name: sample-crypto-job
                        image: docker.artifactory.xxx.com/abc/def/yyz:master-b1b347a

Sensor is getting triggered correctly but is failing to create the job. Can someone please help, what am I missing?

Answer 1

Just ran into the same issue in argo-events. Hopefully this gets fixed in the near future or at least some better documentation.

Change the following value in your sensor.yaml:

spec.triggers[0].template.k8s.resource: jobs

The relevant documentation (at this moment) seems to be pointing to some old Kubernetes API v1.13 documentation, so I've no idea why this needs to be written in the plural "jobs" but this solved the issue for me.

In the example trigger, where a Pod is triggered, the value "pods" is used the same field which pointed me in the right direction.

Answer 2

Posting this as community wiki for better visibility, feel free to edit and expand it.

The original issue was resolved by adjusting role and giving * verbs. Which means argo sensor requires more permissions in fact.

This is a working solution for testing environment, while for production RBAC should be used with principle of least privileges.

How to test RBAC

There's a kubectl syntax which allows to test if RBAC (service account + role + rolebinding) was set up as expected.

Below is example how to check if SERVICE_ACCOUNT_NAME in NAMESPACE can create jobs in namespace NAMESPACE:

kubectl auth can-i --as=system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT_NAME create jobs -n NAMESPACE

The answer will be simple: yes or no.

Usefull links:

• Using RBAC authorization
• Checking API access