CODEWITHSUNDEEP
google-kubernetes-enginegcloud
SST
SST

Reputation: 2144

Getting "Insufficient Permission" error while storing data into GCS from docker container in kubernetes engine

I have configured gcloud with the project, service account and other configurations. Enabled google api and runs the application on GKE clusters.

I am getting the following error logs,

Exception in thread "Thread-10" com.google.cloud.storage.StorageException: Insufficient Permission
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:227)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:308)
    at com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:203)
    at com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:200)
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
    at com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:199)
    at com.google.cloud.storage.StorageImpl.create(StorageImpl.java:161)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
POST https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?projection=full&uploadType=multipart
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Insufficient Permission",
    "reason" : "insufficientPermissions"
  } ],
  "message" : "Insufficient Permission"
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:555)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:305)
    ... 12 more

Upvotes: 0

Views: 1061

Answers (2)

mario
mario

Reputation: 11138

this is not an rbac related error. You just don't have access to GCS. May be scopes? Go to any node, and check if your nodes have GCS scope. – suren May 15 at 18:20

It looks like @suren is right and most probably it is the issue related to GCS scopes as the 403 error can be related with authorization problems. You can check the scopes set for particular Compute Engine VM by running on it:

curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/?recursive=true" -H "Metadata-Flavor: Google"

It will retrieve this information from metadata server. You will probably see only storage-ro scope when it comes to GCS as it is the default setting:

"scopes":["https://www.googleapis.com/auth/devstorage.read_only"

and as you can see here, you will need to set:

storage-rw  https://www.googleapis.com/auth/devstorage.read_write

which on Compute Engine instances can be done as follows:

$ gcloud beta compute instances set-scopes example-instance \
      --scopes=storage-rw --zone=us-central1-b \
      --service-account=example-account

It requires however shutting down the instance to be able to apply changes to service-account. Otherwise you'll get the following error message:

ERROR: (gcloud.beta.compute.instances.set-scopes) Could not fetch resource:
 - The instance must be stopped before the service account can be changed.

But when using GKE you cannot simply stop individual worker node as it is part of a managed instance group.

However you can set scopes when creating new GKE cluster e.g. like this:

gcloud container clusters create test --scopes=storage-rw --service-account default --zone europe-west3-c

Upvotes: 0

DazWilkin
DazWilkin

Reputation: 40326

IIUC, have a look at this:

https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform

You need to create a Kubernetes (!) secret representing the GCP service account credentials.

You then volume-mount the secret into the containers that need to use the secret and set GOOGLE_APPLICATION_CREDENTIALS referencing the account in the container's environment.

Upvotes: 1

Related Questions