CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-storagegoogle-cloud-datalab
sometimesiwritecode
sometimesiwritecode

Reputation: 3213

How would I loop through all the file names in a subdirectory on Google Cloud Storage with python?

Say I have some bucket/subdirectory on Google Cloud Storage and this bucket's address is:

gs://test-monkeys-example/training_data/cats

In this cats subdirectory I have a bunch of images of cats, all of which are jpgs. How would I in python loop through the cats subdirectory and print out all the names of the files in it?

Something like:

for x in directory('gs://test-monkeys-example/training_data/cats'):
    print(x)

Obviously directory('gs://test-monkeys-example/training_data/cats') is not how to do this and is just psuedocode- how would i do this?!

Upvotes: 7

Views: 12956

Answers (2)

Brandon Yarbrough
Brandon Yarbrough

Reputation: 38379

Google Cloud Storage supports listing only objects that begin with a certain prefix. You can access it from the client library like so:

from google.cloud import storage

client = storage.Client()
bucket = client.bucket('mybucket')
for blob in bucket.list_blobs(prefix='training_data/cats'):
  print blob.name

Upvotes: 14

yelsayed
yelsayed

Reputation: 5532

Use the storage module:

import google.datalab.storage as storage
cats = [o.key for o in storage.Bucket('test-monkeys-example').objects()
  if o.key.startswith('training_data/cats')]

This gives you a list of such cats.

Alternatively, you could use the Objects class:

cats = [o.key for o in storage.Objects('test-monkeys-example', '', '')
  if o.key.startswith('training_data/cats')]

If you don't need the list put in a variable, you can use the %gcs magic, it's easier:

%gcs list -o gs://test-monkeys-example/training_data/cats/*

This prints an HTML table of the keys. Note this is a full GCS path, starting with gs://.

Upvotes: 7

Related Questions