CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
azharmalik3
azharmalik3

Reputation: 563

How to get total objects and total storage of each folder on S3 bucket

I have an S3 bucket named camera-assets, in that bucket I have a separate folder for each camera and each folder contains millions / billions of files. I want to get a total number of files and storage of each folder in my bucket.

I tried following CLI command:

aws s3api list-objects --bucket camera-assets --prefix 1011-front-external/snapshots/ --output json --query "[sum(Contents[].Size), length(Contents[])]"

But failed to get the result because folder has million/billion of files. It works if there are only a few files.

Please help me to get files count and storage.

Upvotes: 4

Views: 10169

Answers (2)

Ashish-G
Ashish-G

Reputation: 417

AWS command line interface tool has --summarize option (have to use along with --recursive), which shows number of objects & total size:

$ aws s3 ls s3://camera-assets --summarize --recursive
   ...
Total Objects: xx
   Total Size: xx

Upvotes: 7

Frederic Henri
Frederic Henri

Reputation: 53703

I dont have a bucket with your amount of data to give a try but I found s4cmd to be quite good in the past

you can run commands as

$ s4cmd du s3://camera-assets
$ s4cmd du s3://camera-assets/1011-front-external/snapshots/

What you can do is also looking at cloudwatch - select the S3 metrics, for each bucket you have 2 metrics: - NumberOfObjects - NumberOfObjects

benefit is that you will see how it evolves over time - drawback is that its only at bucket level so you cannot get by folder level.

Upvotes: 0

Related Questions