CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-functionsgoogle-cloud-storage
Data Science Acolyte
Data Science Acolyte

Reputation: 333

Uploading Multiple Web Files to Cloud Storage With Python

I am trying to upload multiple web files to a storage bucket using python. I have a service account set up to enable the credentials and so it should be working. However, every time I try to run this code I receive this error:

Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/voterfile-oh?projection=noAcl: [email protected] does not have storage.buckets.get access to voterfile-oh.

Can I receive help on this issue? I was not able to gather any helpful information from this question nor did the questioner receive an answer that resolved the issue.

def upload_voterfile(bucket_name):
    from google.cloud import storage
    from google.oauth2.service_account import Credentials
    import os
    import json

    credentials_dict = {credentials}
    with open('credentials.json', 'w') as json_file:
        json.dump(credentials_dict, json_file)

    credentials = Credentials.from_service_account_file('credentials.json')

    client = storage.Client(credentials=credentials, project='oh-data-pipeline')
    bucket = client.get_bucket(bucket_name)
    county_list= ['Adams','Allen', 'Ashland', 'Ashtabula', 'Athens', 'Auglaize', 'Belmont', 'Brown', 'Butler', 'Carroll', 
                  'Champaign', 'Clark', 'Clermont', 'Clinton', 'Columbiana', 'Coshocton', 'Crawford', 'Cuyahoga', 'Darke', 
                  'Defiance', 'Delaware', 'Erie', 'Fairfield', 'Fayette', 'Franklin', 'Fulton', 'Gallia', 'Geauga', 'Greene', 
                  'Guernsey', 'Hamilton', 'Hancock', 'Hardin', 'Harrison', 'Henry', 'Highland', 'Hocking', 'Holmes', 
                  'Huron', 'Jackson', 'Jefferson', 'Knox', 'Lake', 'Lawrence', 'Licking', 'Logan', 'Lorain', 'Lucas', 'Madison',
                  'Mahoning', 'Marion', 'Medina', 'Meigs', 'Mercer', 'Miami', 'Monroe', 'Montgomery', 'Morgan', 'Morrow', 
                  'Muskingum', 'Noble', 'Ottawa', 'Paulding', 'Perry', 'Pickaway', 'Pike', 'Portage', 'Preble', 'Putnam', 
                  'Richland', 'Ross', 'Sandusky', 'Scioto', 'Seneca', 'Shelby', 'Stark', 'Summit', 'Trumbull', 'Tuscarawas', 
                  'Union', 'Van Wert', 'Vinton', 'Warren', 'Washington', 'Wayne', 'Williams', 'Wood', 'Wyandot']

    for f, g in zip(range(1, 89), county_list):
        file = urllib.request.urlopen('https://www6.sos.state.oh.us/ords/f?p=VOTERFTP:DOWNLOAD::FILE:NO:2:P2_PRODUCT_NUMBER:{}'.format(f))
        blob = bucket.blob('{}'.format(g))
        blob.upload_from_string(data=file.read(), content_type="text/plain")

if __name__ == "__main__":

    upload_voterfile(bucket_name='voterfile-oh')

Upvotes: 2

Views: 492

Answers (1)

Happy-Monad
Happy-Monad

Reputation: 2002

I have been able to reproduce your issue, here are the steps I have followed:

  1. Created a service account and assign Storage Object Creator permission.
  2. Ran gcloud iam service-accounts keys create [FILE_NAME].json --iam-account [NAME]@[PROJECT_ID].iam.gserviceaccount.com to get account credentials file.
  3. Erased lines 7 to 9 of your script and changed the filename of line 11 to match the name of the downloaded credentials file.
  4. Ran the script. Here I get the same error as you.

The reason behind this error is that on line 14 you're getting your bucket object through get_bucket method. This method queries cloud Storage, requiring get permissions on your bucket but the Storage Object Creator role does not include get permissions.

To solve your issue just change line 14 with this code bucket = client.bucket(bucket_name) which directly creates a bucket object without interacting with Cloud Storage and therefore it does not raise the permission error, see corresponding reference.

Another solution might be to change the service account role to Storage Object Admin because it includes get permissions.

Upvotes: 0

Related Questions