CODEWITHSUNDEEP
pythonjsongoogle-cloud-platformgoogle-cloud-storagehttp-error
sportzcode
sportzcode

Reputation: 125

How can I resolve "HttpError: Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object., 401" error?

The problem: I'm trying to read in a .gz JSON file that is stored in one of my project's cloud storage bucket using a google colab python notebook and I keep getting this error:

HttpError: Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object., 401

My code:

fs = gcsfs.GCSFileSystem(project='my-project')
with fs.open('bucket/path.json.gz') as f:
    gz = gzip.GzipFile(fileobj=f) 
    file_as_string = gz.read()
    json_a = json.loads(file_as_string)

I've tried all of these authentication methods and still get the same 401 error :

!gcloud auth login
!gcloud auth list
!gcloud projects list
!gcloud config set project 'myproject-id'

from google.colab import auth
auth.authenticate_user()

!gcloud config set account 'my GCP email'

!gcloud auth activate-service-account

!gcloud auth application-default login

!gsutil config

!gcloud config set pass_credentials_to_gsutil false

!gsutil config -a

I've also set my GCP IAM permissions to:

Upvotes: 11

Views: 23594

Answers (1)

DazWilkin
DazWilkin

Reputation: 40366

It's not entirely clear from your question but:

  1. gcloud and Google SDKs both use Google's identity|auth platform but they don't share state. You usually (!) can't login using gcloud and expect code using an SDK to be authenticated too
  2. @john-hanley correctly points out that one (often confusing) way to share state between gcloud and code using Google SDKs is to use gcloud auth application-default login. However, this only works because gcloud writes its state locally and code using Google SDKs when running as the same user on the same host, will be able to access this state. I think (!?) this won't work with browser-based collab
  3. I'm unfamiliar with gcsfs.GCSFileSystem but, it is not a Google SDK. Unless its developers have been particularly thoughtful, it won't be able to leverage authentication done by the Google SDK using auth.authenticate_user().

So...

I think you should:

  1. Ensure that your user account ([email protected] or whatever) has roles/storage.objectAdmin (or any predefined role that permits storage.objects.get).
  2. Use google.collab.auth and auth.authenticate_user() to obtain credentials for the browser's logged-in user (i.e. [email protected]).
  3. Use a Google Cloud Storage library, e.g. google-cloud-storage to access the GCS object. The Google library can leverage the credentials obtained in the previous step.

Update

Here's an example.

NOTE: it use the API Client Library rather than the Cloud Client Library but these are functionally equivalent.

Upvotes: 2

Related Questions