CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-sdk
matt burns
matt burns

Reputation: 25380

Cheapest way to delete 2 billion objects from S3 IA

I have a bucket in S3 (Infrequent access) containing 2 billion objects. It is too big to delete in the console or over the api without taking years.

I can create a lifecycle rule to expire and delete the objects but the calculator predicts this will cost me >$20,000. Is that correct? Is there a better way to delete a bucket?

I have a file effectively containing a list of all the objects in that bucket if that helps.

Update 2021:

An answer below from @MAP points out that there is now an "Empty" button. I haven't tested yet, but looks like the way to go (I'll accept that answer once tested):

screenshot of empty button

Upvotes: 25

Views: 19045

Answers (4)

PencilBow
PencilBow

Reputation: 1078

Expiration lifecycle rules are free. From the original feature announcement:

As with standard delete requests, Amazon S3 doesn’t charge you for using Object Expiration.

Upvotes: 7

MAP
MAP

Reputation: 151

In 2021, anyone who comes across this question may benefit to know that AWS console now provides an empty button.

Select the bucket and click on "empty" button and all objects versioned or not versioned would be emptied/deleted. Depending on the number of objects it can take minutes to days.

Upvotes: 15

Sébastien Stormacq
Sébastien Stormacq

Reputation: 14905

Delete operations are for free. You can create a lifecycle Policy to automate a bulk delete.

I would start with a small number of objects first and check billing report to 100% confirm that the delete will not be charged, then go for the rest.

Upvotes: 4

A.Khan
A.Khan

Reputation: 3992

If you have a list of all the objects available then you can certainly use Multi Delete Object action. Apparently this API is free. I would create AWS Step Functions state machine to loop through the file and delete 1000 objects at a time. 1000 appears to be the limit.

It will take around 2M step function transactions to delete all the objects in the bucket. As per the pricing for step function it will cost you around $50 + cost of Lambda invocations around $1 so total cost roughly $51.

Update

Using Lambda or Step Functions is probably not the most cost effective option because both ways you will need to read the file (that contains object keys) from some source such as S3. So I think running the script from local machine or any EC2 linux screen appears to be the best option.

Upvotes: 18

Related Questions