Gcloud permission error when uploading big files with service account

Question

I'm trying to upload a gzipped postgres dump file to gcp storage. I'm running this in a docker container with image google/cloud-sdk:alpine(I've also tried local gcloud installation):

gcloud storage mv file.dump.gz gs://my-bucket/

I'm using a service account authentication for gcloud by pointing CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE environment variable to the json key file(I've also tried GOOGLE_APPLICATION_CREDENTIALS and GOOGLE_GHA_CREDS_PATH).

The service account has "Storage Object Admin" role.

The bucket has "Nearline" storage class with a lifecycle deletion of 5days.

When I try to upload a small file(largest that I tested that works is 14MB) it works, but when I try to upload a big file(smallest that I tested that fails is 201MB) it fails with error

ERROR: (gcloud.storage.mv) User [None] does not have permission to access b instance [my-bucket] (or it may not exist): does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).

How do I upload big files without getting permission error?

Answer 1

gcloud uses a mechanism called Parallel Composite Uploads to speed up file transfers to GCS, described here: https://cloud.google.com/storage/docs/parallel-composite-uploads

gcloud checks the compatibility of the bucket's default storage class and retention period with parallel composite uploads before using it. Therefore it performs a GET request to the bucket. This request requires the storage.buckets.get permission on the service account of the caller.

Option 1

Add a role such as roles/storage.admin to the service account, which includes the storage.buckets.get permission.

https://cloud.google.com/storage/docs/access-control/iam-roles

Option 2

Disable gcloud's use of parallel composite uploads:

gcloud config set storage/parallel_composite_upload_enabled false

https://cloud.google.com/sdk/gcloud/reference/topic/configurations#parallel_composite_upload_enabled

Option 3

Disable the compatibility check:

gcloud config set storage/parallel_composite_upload_compatibility_check false

https://cloud.google.com/sdk/gcloud/reference/topic/configurations#parallel_composite_upload_compatibility_check

These are the possible solutions I found, the list might not be exhaustive.

Answer 2

It seems you need "Storage Admin" role to upload big files!

For anyone testing note that adding permissions(for example going from "Storage Object Admin" to "Storage Admin") seems to happen instantly where as removing permissions(for example going from "Storage Admin" to "Storage Object Admin") might take a few minutes to be applied.