CODEWITHSUNDEEP
google-cloud-platformgoogle-kubernetes-engine
Ire00
Ire00

Reputation: 131

Access to Google Cloud Storage from an Autopilot GKE cluster

I am having issues querying Google Cloud Storage from an Autopilot GKE cluster. Previously, when the cluster was standard, I had no problems (as long as the cluster was created with: --scopes “https://www.googleapis.com/auth/cloud-platform”).

Now I need to use an Autopilot cluster instead of the Standard and the error i get on the GKE cluster is like this:

google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/my-bucket?projection=noAcl&prettyPrint=false: Caller does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).

In addition to the scope, I have associated a service account which can query GCS and BQ. I already use the same service account in a compute engine VM with no problems. However on the Autopilot cluster I keep having this problem.

Anyone have any suggestions?

Upvotes: 2

Views: 1217

Answers (1)

Gari Singh
Gari Singh

Reputation: 12033

Autopilot clusters always enable Workload Identity for accessing other Google Cloud services from your Pods. So you'll need to:

  1. Create a Kubernetes service account in the namespace(s) in which you've deployed Airflow
kubectl create serviceaccount $KSA_NAME -n $NS

(replace $KSA_NAME and $NS accordingly)

  1. Create an IAM service account in your project:
gcloud iam service-accounts create $GSA_NAME \
    --project=$PROJECT_ID

(replace $GSA_NAME and $PROJECT_ID accordingly)

  1. Bind the IAM service account to the required role for accessing your storage bucket(s):
gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
  --member "serviceAccount:$GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=objectViewer

(replace $BUCKET_NAME with your bucket and you can use a different role is you require more than get on the bucket)

  1. Allow the Kubernetes service account to impersonate the GSA:
gcloud iam service-accounts add-iam-policy-binding $GSA_ NAME@$PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[$NS/$KSA_NAME]"
  1. Annotate your Kubernetes service account
kubectl annotate serviceaccount $KSA_NAME \
    --namespace $ns \
    iam.gke.io/gcp-service-account=$GSA_ NAME@$PROJECT_ID.iam.gserviceaccount.com
  1. Use the GSA in your deployment spec:
...
spec:
  serviceAccountName: $KSA_NAME
...

See Configure applications to use Workload Identity for more details.

Upvotes: 2

Related Questions