CODEWITHSUNDEEP
python-3.xgoogle-cloud-storage
codebrotherone
codebrotherone

Reputation: 581

How do you iterate through objects in a blob on google cloud storage? Python

I am trying to figure out how to iterate over objects in a blob in google cloud storage. The address is similar to this:

gs://project_ID/bucket_name/DIRECTORY/file1 
gs://project_ID/bucket_name/DIRECTORY/file2
gs://project_ID/bucket_name/DIRECTORY/file3
gs://project_ID/bucket_name/DIRECTORY/file4
...

The DIRECTORY on the GCS bucket has a bunch of different files that I need to iterate over, so that I can check when it was last updated (to see if it is a new file there) so that I can pull the contents.

Example function

def getNewFiles():
    storage_client = storage.Client(project='project_ID')
    try:
        bucket = storage_client.get_bucket('bucket_name')
    except:
        storage_client.create_bucket(bucket_name)

    for blob in bucket.list_blobs(prefix='DIRECTORY'):
        if blob.name == 'DIRECTORY/':
            **Iterate through this Directory**
            **CODE NEEDED HERE***
            **Figure out how to iterate through all files here**

I have gone through the python api and the client library, and can't find any examples of this working..

Upvotes: 1

Views: 8286

Answers (1)

VictorGGl
VictorGGl

Reputation: 1916

According to Google Cloud Client Library for Python docs, blob.name:

This corresponds to the unique path of the object in the bucket

Therefore blob.name will return something like this:

DIRECTORY/file1

If you are already including the parameter prefix='DIRECTORY' when using the list_blobs() method you can get all your files in your directory by doing:

for blob in bucket.list_blobs(prefix='DIRECTORY'):
    print(blob.name)

You can use something like blob.name.lstrip('DIRECTORY') or the standard library re module to clean the string and get only the file name.

However, according to what you said: "so that I can check when it was last updated (to see if it is a new file there)" if you are looking for some function to be triggered when you have new files in your bucket, you can use Google Cloud Functions. You have the docs here on how to use them with Cloud Storage when new objects are created. Although as of current date (Feb/2018) you can only write Cloud Functions using NODE.JS

Upvotes: 2

Related Questions