CODEWITHSUNDEEP
pythonamazon-s3boto3
Anshul Srivastava
Anshul Srivastava

Reputation: 37

Python - Search for multiple files in subfolder present inside a bucket using boto3

So I have a requirement of downloading all the JSON files that are uploaded to the subfolder present inside the bucket.

Is there any way to do so?

I tried the following way:

for key in bucket_name.objects.filter(Prefix='report'):
        print key
        s3_client.download_file(key, bucket_name, '{}/{}'.format(current_directory, key))

And I'm getting the following error on doing so:

AttributeError: 'str' object has no attribute 'objects'

Upvotes: 1

Views: 1128

Answers (2)

Anshul Srivastava
Anshul Srivastava

Reputation: 37

On listing the files present in bucket, i was getting output as -

s3.ObjectSummary(bucket_name='', key=u'file_Searching_for')

After trying multiple options i came upto the following code-

def main():

    objs = bucket_name.objects.filter(Prefix=Prefix_name)

    for obj in objs:
        filename = obj.key
        if filename.endswith('.json'):
           <download the file if you want to>

From the above code, it will give you the file name with the prefix, which you can split and download the file. It will help you to download any number of files that are present in the bucket which ends with .json

Let me know if the answer helps you. It helped me out.

Thanks

Upvotes: 0

Marcin
Marcin

Reputation: 238179

Unfortunately, S3 does not have a feature to list objects in a bucket by a suffix, such as .json. The only way is to iterate over the objects on the client side and filter them programmatically.

You can develop your own python script for that, as you already started. But this is such a common operation that that there are many recipes already written for that. Some of them for bash and python can be found here.

These solutions are good for ad hoc analysis. But if you have lots of files and have to search for them regularly, it can be time-consuming. One solution to that would be to use S3 Inventory to generate a csv list of your objects on daily or weakly basis, and then have your python to use the csv file generated to identify files of interest.

You could even fully automate this process, as you can get notified when the inventory is available and trigger lambda which will filter the csv file.

Upvotes: 1

Related Questions