CODEWITHSUNDEEP
pythonboto3
Djikii
Djikii

Reputation: 157

How to filter Boto3 s3 objects?

I want to change this line

   files = os.listdir('/Users/milenko/mario/Json_gzips')

in my code,to read .gz files from my bucket straight into list. I tried

>>> import boto3
>>> s3 = boto3.resource('s3')
>>> s3
s3.ServiceResource()
>>> my_bucket = s3.Bucket('cw-dushpica-tests')

>>> for object_summary in my_bucket.objects.filter(Prefix='*.gz'):
...     print(object_summary)

There is no output,it does print nothing.

for object_summary in my_bucket.objects.filter(Prefix='/'):
...     print(object_summary)

The same,got nothing.

How should my prefix look like?

Upvotes: 1

Views: 11113

Answers (1)

Lamanus
Lamanus

Reputation: 13541

The prefix parameter of the filter method means that

Prefix (string) -- Limits the response to keys that begin with the specified prefix.

So, you can limit the path to the specific folder and then filter by yourself for the file extension.

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('your_bucket')

keys = []

for obj in bucket.objects.filter(Prefix='path/to/files/'):
    
    if obj.key.endswith('gz'):
        
        keys.append(obj.key)
        
print(keys)

Upvotes: 5

Related Questions