CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
dnk.nitro
dnk.nitro

Reputation: 496

Bulk AWS S3 files exist verification?

What is the best/fastest approach to check if multiple files exist in AWS S3 bucket?

For example I have 100k files metadata in my local DB. I would like to make sure all of them exist in S3 bucket. I can do 'aws s3 ls' for particular file, but that would mean 100k aws requests. Is there a better approach to this?

Upvotes: 4

Views: 2784

Answers (3)

Alex Tamoykin
Alex Tamoykin

Reputation: 185

If you would like to make sure your local files are on S3 you can try the s3 sync command.

You can also check out which files are there currently with Commandeer, which supports S3 file browsing in a nice tree view.

enter image description here

Upvotes: 1

John Rotenstein
John Rotenstein

Reputation: 269101

If you are just doing a general audit, you could use Amazon S3 Inventory to obtain a complete daily dump of all object keys and associated metadata.

You could then write some code to compare the contents of the Inventory file against the DB entries.

Upvotes: 3

Matus Dubrava
Matus Dubrava

Reputation: 14462

If you want to retrieve all keys in a specific bucket in one command then you can use this.

aws s3api list-objects --bucket <bucket-name> --no-paginate

Once you have that list, you can process it by a custom code.

Upvotes: 2

Related Questions