GCP GKE: Service account "is attempting to grant RBAC permissions not currently held"

Question

I'm setting up a CI/CD pipeline for deploying a Kubernetes-based application in an automated way. Part of this deployment involves creating other service accounts and their associated roles.

When my pipeline runs, deployment fails with this error message:

Error: roles.rbac.authorization.k8s.io "mongodb-kubernetes-operator" is forbidden: user "cicd-bot@my-project.iam.gserviceaccount.com" (groups=["system:authenticated"]) is attempting to grant RBAC permissions not currently held:
│ {APIGroups:[""], Resources:["configmaps"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:[""], Resources:["pods"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:[""], Resources:["secrets"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:[""], Resources:["services"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:["apps"], Resources:["statefulsets"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:["mongodbcommunity.mongodb.com"], Resources:["mongodbcommunity"], Verbs:["list" "watch" "update" "patch" "get"]}
│ {APIGroups:["mongodbcommunity.mongodb.com"], Resources:["mongodbcommunity/finalizers"], Verbs:["list" "watch" "update" "patch" "get"]}
│ {APIGroups:["mongodbcommunity.mongodb.com"], Resources:["mongodbcommunity/spec"], Verbs:["list" "watch" "update" "patch" "get"]}
│ {APIGroups:["mongodbcommunity.mongodb.com"], Resources:["mongodbcommunity/status"], Verbs:["list" "watch" "update" "patch" "get"]}
│ 
│   with module.db_document.kubernetes_role.operator_mongodb,
│   on modules/db_document/main.tf line 17, in resource "kubernetes_role" "operator_mongodb":
│   17: resource "kubernetes_role" "operator_mongodb" {
│ 

The error seemed straightforward enough: my service account can't grant permissions that it does not have. Since the error message mentioned my GCP service account, cicd-bot@my-project.iam.gserviceaccount.com, I added to my role definition what I believed to be the matching permissions.

Below is my resulting role. It has create, delete, get, list, and update permissions for configMaps, pods, secrets, services, statefulsets , and thirdPartyObjects, which I believed should cover the requirements.

resource "google_project_iam_custom_role" "cicd_bot_role" {
  project = var.project
  role_id = "cicd_bot"
  title   = "CICD Bot"
  permissions = [
    "artifactregistry.repositories.downloadArtifacts",
    "artifactregistry.repositories.uploadArtifacts",
    "compute.instanceGroupManagers.get",
    "container.clusters.get",
    "container.configMaps.create",
    "container.configMaps.delete",
    "container.configMaps.get",
    "container.configMaps.list",
    "container.configMaps.update",
    "container.cronJobs.create",
    "container.cronJobs.delete",
    "container.cronJobs.get",
    "container.cronJobs.update",
    "container.customResourceDefinitions.create",
    "container.customResourceDefinitions.delete",
    "container.customResourceDefinitions.get",
    "container.customResourceDefinitions.list",
    "container.customResourceDefinitions.update",
    "container.deployments.create",
    "container.deployments.delete",
    "container.deployments.get",
    "container.deployments.update",
    "container.ingresses.create",
    "container.ingresses.delete",
    "container.ingresses.get",
    "container.ingresses.update",
    "container.jobs.create",
    "container.jobs.delete",
    "container.jobs.get",
    "container.jobs.update",
    "container.namespaces.get",
    "container.persistentVolumeClaims.create",
    "container.persistentVolumeClaims.delete",
    "container.persistentVolumeClaims.get",
    "container.persistentVolumeClaims.update",
    "container.pods.create",
    "container.pods.delete",
    "container.pods.get",
    "container.pods.list",
    "container.pods.update",
    "container.roleBindings.create",
    "container.roleBindings.delete",
    "container.roleBindings.get",
    "container.roleBindings.update",
    "container.roles.create",
    "container.roles.delete",
    "container.roles.get",
    "container.roles.update",
    "container.secrets.create",
    "container.secrets.delete",
    "container.secrets.get",
    "container.secrets.list",
    "container.secrets.update",
    "container.serviceAccounts.create",
    "container.serviceAccounts.delete",
    "container.serviceAccounts.get",
    "container.serviceAccounts.update",
    "container.services.create",
    "container.services.delete",
    "container.services.get",
    "container.services.list",
    "container.services.update",
    "container.statefulSets.create",
    "container.statefulSets.delete",
    "container.statefulSets.get",
    "container.statefulSets.list",
    "container.statefulSets.update",
    "container.thirdPartyObjects.create",
    "container.thirdPartyObjects.delete",
    "container.thirdPartyObjects.get",
    "container.thirdPartyObjects.list",
    "container.thirdPartyObjects.update",
    "dns.changes.create",
    "dns.changes.get",
    "dns.resourceRecordSets.get",
    "dns.resourceRecordSets.list",
    "dns.resourceRecordSets.update",
    "storage.buckets.get",
    "storage.objects.create",
    "storage.objects.delete",
    "storage.objects.get",
    "storage.objects.list",
  ]
}

However, after deploying this, the error remained the same. I wondered if adding equivalent permissions on the kubernetes side was necessary, so I created the following ClusterRole and ClusterRoleBinding, too.

resource "kubernetes_cluster_role" "cicd_bot" {
  metadata {
    name = kubernetes_service_account.cicd_bot.metadata[0].name
  }
  rule {
    api_groups = [""]
    resources  = ["namespaces"]
    verbs      = ["create", "delete", "get"]
  }
  rule {
    api_groups = [""]
    resources  = ["configmaps"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["pods"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["secrets"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["services"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = ["apps"]
    resources  = ["statefulsets"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = ["mongodbcommunity.mongodb.com"]
    resources  = ["mongodbcommunity"]
    verbs      = ["list", "watch", "update", "patch", "get"]
  }
  rule {
    api_groups = ["mongodbcommunity.mongodb.com"]
    resources  = ["mongodbcommunity/finalizers"]
    verbs      = ["list", "watch", "update", "patch", "get"]
  }
  rule {
    api_groups = ["mongodbcommunity.mongodb.com"]
    resources  = ["mongodbcommunity/spec"]
    verbs      = ["list", "watch", "update", "patch", "get"]
  }
  rule {
    api_groups = ["mongodbcommunity.mongodb.com"]
    resources  = ["mongodbcommunity/status"]
    verbs      = ["list", "watch", "update", "patch", "get"]
  }
}

resource "kubernetes_cluster_role_binding" "cicd_bot" {
  metadata {
    name = kubernetes_service_account.cicd_bot.metadata[0].name
  }
  subject {
    kind      = "ServiceAccount"
    namespace = kubernetes_service_account.cicd_bot.metadata[0].namespace
    name      = kubernetes_service_account.cicd_bot.metadata[0].name
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.cicd_bot.metadata[0].name
  }
}

Unfortunately, the pipeline still fails with the same error. I've been able to overcome a similar error in the past, but not this time. What am I missing?

UPDATE: I was able to deploy successfully by attaching the role roles/container.admin to my service account. So now I need to figure out which permission roles/container.admin has that my custom role does not.

Answer 1

Solved my issue:

  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  - clusterroles
  - roles
  verbs:
  - bind
  - escalate
  - create
  - delete

Answer 2

I had to give the Google Service Acccount cluster-admin clusterrole in the K8s cluster using RBAC:

kubectl create clusterrolebinding $SERVICE_ACCOUNT_NAME \
  --clusterrole cluster-admin \
  --user $SERVICE_ACCOUNT_NAME@$PROJECT_ID.iam.gserviceaccount.com

You don't need the container.roles.escalate permission anymore after you do that. I only gave myself the GKE Cluster Developer predefined IAM role.

Answer 3

Sadly, the one permission that was missing was

container.roles.escalate

Even including every other container.* permission was insufficient; container.roles.escalate was still needed.

This is unfortunate because it makes the cluster more vulnerable to permissions escalation attacks. If there is a safer way to accomplish this, I'd love to hear it. I won't mark my own answer as "correct" because I'm not satisfied with it. But hey, at least it's working...