CODEWITHSUNDEEP
google-cloud-storage
jamiet
jamiet

Reputation: 12364

gsutil cp succeeds but upload_from_filename() method from storage client fails

I want to upload a file to GCS using the google storage client in python but its failing with a permissions issue, however gsutil cp succeeds. I can't understand why.

Here is what I have run using gsutil:

BUCKET=abc
$ gcloud iam service-accounts keys create --iam-account $ACCOUNT key_file.json
created key [5006838b5984f1d3b4de6523239e9bbd2c7f7047] of type [json] as [key_file.json] for [[email protected]]
$ gcloud auth activate-service-account --key-file key_file.json
Activated service account credentials for: [[email protected]]
$ touch test.txt
$ gsutil cp test.txt gs://${BUCKET}/test.txt     
Copying file://test.txt [Content-Type=text/plain]...
/ [1 files][    0.0 B/    0.0 B]                                                
Operation completed over 1 objects.

Here is what I have attempted from python:

from google.cloud import storage
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'key_file.json' #same file as downloaded above
storage_client = storage.Client()
bucket_name = "abc" #the bucket name
source_file_name = "test.txt"
destination_blob_name = f"{source_file_name}"
with open(source_file_name, 'w') as f:
        f.write("lorem ipsum")
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_filename(source_file_name)

That upload_from_filename() function fails with error:

google.api_core.exceptions.Forbidden: 403 POST https://storage.googleapis.com/upload/storage/v1/b/abc/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)

I'm baffled. I would have thought that if this worked from gsutil it would work using the python storage client. Any suggestions would be most welcome.

Upvotes: 0

Views: 946

Answers (2)

jamiet
jamiet

Reputation: 12364

I've gotten to the bottom of this problem. It was twofold.

  1. I discovered that the code would succeed if the storage object destination_blob_name did not already exist. If the object did exist then the error explained above would occur.
  2. Permission for $ACCOUNT to access the bucket was provided by a custom role that was applied to the bucket. When I added permission storage.objects.get & storage.objects.delete to that custom role then my code succeeded, even if the object already existed.

I've discovered that storage.objects.get & storage.objects.delete are described as

  • Read object data and metadata, excluding ACLs
  • Delete objects

https://cloud.google.com/storage/docs/access-control/iam-permissions#object_permissions

The learning I've taken away from this is that under certain circumstances (e.g. when the destination object already exists) then gsutil cp requires different permissions to google.cloud.storage.blob.upload_from_filename(). If anyone out there can elucidate the difference between those two operations then I'd love to understand it better.

Upvotes: 3

Harif Velarde
Harif Velarde

Reputation: 753

This error (403) indicates that the user was not authorized by Google Cloud Storage to make the request, basically you need to keep in mind that:

Upvotes: 0

Related Questions