CODEWITHSUNDEEP
google-cloud-platformgoogle-secret-managerexternal-secrets-operator
Steven Marks
Steven Marks

Reputation: 704

Google Secret Manager with GKE and external secrets error: could not get idbindtoken

I'm in the process of moving to secret manager with external secrets operator in our GKE Standard Cluster.

I have been getting the below error and I am struggling to see why this is happening:

failed to create GCP secretmanager client: unable to fetch identitybindingtoken: could not get idbindtoken token, status: 404

What is causing this and how to troubleshoot it?

terraform:

resource "google_service_account" "external_secret_sa" {
  account_id  = "external-secrets-sa"
  description = "Service Account for external secrets"
  project     = var.project_id
}

# Gives the external secrets Service account access to token creator
resource "google_project_iam_binding" "external_secret_sa_token_creator" {
  project = var.project_id
  role    = "roles/iam.serviceAccountTokenCreator"
  members = [
    "serviceAccount:${google_service_account.external_secret_sa.email}"
  ]
}
# Gives the external secrets Service account access to the secret manager. 
resource "google_project_iam_binding" "external_secret_sa_access" {
  project = var.project_id
  role    = "roles/secretmanager.secretAccessor"
  members = [
    "serviceAccount:${google_service_account.external_secret_sa.email}"
  ]
}

#Allow kubernetes service account to impersonate GCP service account
resource "google_service_account_iam_binding" "external_secret_sa_bind" {
  service_account_id = google_service_account.external_secret_sa.name
  role               = "roles/iam.workloadIdentityUser"

  members = [
    "serviceAccount:${var.project_id}.svc.id.goog[external-secrets/${google_service_account.external_secret_sa.account_id}]"
  ]
  depends_on = [google_service_account.external_secret_sa]
}

helm chart config for external-secrets (its a dependency in another chart)

external-secrets:
  installCRDs: true
  serviceAccount:
    create: true
    name: external-secrets-sa
    annotations:
      iam.gke.io/gcp-service-account: "[email protected]"

ClusterSecretStore:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-store
spec:
  provider:
    gcpsm:
      projectID: {{ .Values.projectId }}
      auth:
        workloadIdentity:
          # name of the cluster Location, region or zone
          clusterLocation: {{ .Values.region }}
          # name of the GKE cluster
          clusterName: {{ .Values.gkeCluster }}
          # reference the sa from above
          serviceAccountRef:
            name: {{ index .Values "external-secrets" "serviceAccount" "name" }}
            namespace: {{ .Release.Namespace }}

GKE pool oauth scope:

node_pools_oauth_scopes = {
    all = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/cloud-platform"

    ]
  }

I feel like the binding between GKE and workload identity SA isn't working as it should, but im not sure if its an issue with GKE cluster or somewhere else.

Upvotes: 0

Views: 660

Answers (1)

Fab T
Fab T

Reputation: 1

I got the same issue when I was running the secret store with an incorrect cluster name defined. I would update your config such as :

  • adding ns where secret store will run ( in metadata )
  • adding ProjectID where secret store will run ( in wokrloadIdentity )

Note that first ProjectID ref is where is GCPSM is running, second is where Secret Store is running, which can be different.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-store
  namespace: {{ .Release.Namespace }}
spec:
  provider:
    gcpsm:
      projectID: {{ .Values.projectId }}
      auth:
        workloadIdentity:
          # name of the cluster Location, region or zone
          clusterLocation: {{ .Values.region }}
          # name of the GKE cluster
          clusterName: {{ .Values.gkeCluster }}
          #name of the ProjectID where SecretStore is running
          projectID: {{ .Values.projectId }}
          # reference the sa from above
          serviceAccountRef:
            name: {{ index .Values "external-secrets" "serviceAccount" "name" }}

Upvotes: 0

Related Questions