CODEWITHSUNDEEP
amazon-web-servicesamazon-s3boto3
scoder
scoder

Reputation: 2611

fast way to empty the s3 bucket using boto3

I am using the below boto3 code (source link) to empty the bucket, but it takes a huge time since the bucket has large numbers of versioned objects to delete.

import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('bucket-name')
bucket.object_versions.delete()

The above code is working but it is taking nearly a day to empty it.

Is there a way to empty it faster?

Upvotes: 1

Views: 902

Answers (1)

Parsifal
Parsifal

Reputation: 4526

If you need to do this synchronously -- you need to know that the bucket is empty when your program is done -- then the only way to speed it up is to spread the work between multiple threads/processes. I would have one main thread that gets a listing of objects and then uses the concurrent.futures package to dispatch a delete on each object in the list.

Be aware that you are charged for all operations invoked from a client program.

If you can do the deletes asynchronously -- over the course of a couple of days -- then the best solution is a lifecycle rule. This happens entirely within S3, and you don't get charged for the individual deletes.

Upvotes: 4

Related Questions