CODEWITHSUNDEEP
machine-learninggoogle-cloud-platformgoogle-cloud-storagegoogle-colaboratorybucket
Taha Mahjoubi
Taha Mahjoubi

Reputation: 402

Error when uploading from a Cloud Storage bucket to Google Colab using 'gsutil'

I've trained a model in a Compute Engine VM instance in GCP and copied the weights into a Cloud Storage bucket using the gsutil cp -r command.

I then made the bucket public and tried to copy those weights into a Google Colab notebook using the command !gsutil cp -r gs://{bucket/folder} ./

However, I get the following error:

ResumableDownloadException: Transfer failed after 23 retries. Final exception: Anonymous caller does not have storage.objects.get access to {folder/path}

Why am I getting this error?

Upvotes: 0

Views: 1344

Answers (1)

Maxim
Maxim

Reputation: 4431

Edit:

  • The Cloud Storage bucket is missing the appropriate Cloud IAM role to make it fully publicly read from. The role roles/storage.objectViewer provides the necessary permissions to read and list objects from the bucket - assigning it to allUsers will make it public.

    Therefore, as per the documentation, this can be achieved with a single gsutil iam command:

    gsutil iam ch allUsers:objectViewer gs://[BUCKET_NAME].

    And then, in in Google Colab you should be able to read (or download) objects from Cloud Storage buckets with:

    !gsutil cp -r gs://[BUCKET_NAME]/[FOLDER_NAME] ./


  • A safer approach is to instead of making the entire Cloud Storage bucket public, to authenticate with it using the following Python code in the notebook:

    from google.colab import auth auth.authenticate_user()

    Then set the project ID you're using with a gcloud command; replacing my-project accordingly:

    !gcloud config set project my-project

    And finally run the gsutil command; replacing bucket and folder:

    !gsutil cp -r gs://[BUCKET_NAME]/[FOLDER_NAME] ./

Upvotes: 3

Related Questions