CODEWITHSUNDEEP
pythonamazon-s3directoryboto3boto
Mikhail Orechkine
Mikhail Orechkine

Reputation: 101

List AWS S3 folders with boto3

I have boto code that collects S3 sub-folders in levelOne folder:

import boto

s3 = boto.connect_s3()
bucket = s3.get_bucket("MyBucket")

for level2 in bucket.list(prefix="levelOne/", delimiter="/"):
    print(level2.name)

Please help to discover similar functionality in boto3. The code should not iterate through all S3 objects because the bucket has a very big number of objects.

Upvotes: 10

Views: 19087

Answers (2)

John Rotenstein
John Rotenstein

Reputation: 269081

If you are simply seeking a list of folders, then use CommonPrefixes returned when listing objects. Note that a Delimiter must be specified to obtain the CommonPrefixes:

import boto3

s3_client = boto3.client('s3')

response = s3_client.list_objects_v2(Bucket='BUCKET-NAME', Delimiter = '/')

for prefix in response['CommonPrefixes']:
    print(prefix['Prefix'][:-1])

If your bucket has a HUGE number of folders and objects, you might consider using Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects.

Upvotes: 16

Marcin
Marcin

Reputation: 238051

I think the following should be equivalent:

import boto3

s3 = boto3.resource('s3') 

bucket = s3.Bucket('MyBucket')

for object in bucket.objects.filter(Prefix="levelOne/", Delimiter="/"):
    print(object.key)

Upvotes: 3

Related Questions