CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
dnivra
dnivra

Reputation: 769

Quickest way to get amazon s3 bucket size

As of now i am calculating the size of an amazon s3 bucket by iterating over all the objects and adding up the size of individual objects. This is quite time consuming. I tried parallelising the operation and saved some time but even then it takes a lot of time.

System I am using : EC2 m1.large

Is there any workarounds or better means to find out the bucket size?

Note: i dont have access to aws console, just have the access keys

Upvotes: 4

Views: 7058

Answers (1)

Jesse Pangburn
Jesse Pangburn

Reputation: 746

Probably a bit late, but I was looking for this too and found you can get this information already computed from CloudWatch. From the command line, you can call this to list the S3 buckets you have which have BucketSizeBytes stat available:

aws cloudwatch list-metrics --metric-name BucketSizeBytes

This gives a list of things like this:

    {
        "Namespace": "AWS/S3", 
        "Dimensions": [
            {
                "Name": "BucketName", 
                "Value": "myReallyGreatBucket"
            }, 
            {
                "Name": "StorageType", 
                "Value": "StandardStorage"
            }
        ], 
        "MetricName": "BucketSizeBytes"
    }

So basically the bucket size is partitioned into Standard storage size and Reduced Redundancy storage size byte counts. You want these separate so you can know how much full price storage you're using and how much of the cheaper storage. If you want the total bytes for another purpose, just add the two. To go get them, you can call it to get a variety of computed-over-time counts AWS CloudWatch get-metric-statistics CLI options. I got it to work by specifying to get the time for a day:

aws cloudwatch get-metric-statistics --namespace AWS/S3 --dimensions Name=BucketName,Value=myReallyGreatBucket Name=StorageType,Value=StandardStorage --metric-name BucketSizeBytes --start-time 2015-08-23T00:00:00 --end-time 2015-08-24T00:00:00 --period 86400 --statistics Average --unit Bytes
{
    "Datapoints": [
        {
            "Timestamp": "2015-08-23T00:00:00Z", 
            "Average": 436306296.0, 
            "Unit": "Bytes"
        }
    ], 
    "Label": "BucketSizeBytes"
}

It didn't work when I tried to get the average for a minute or an hour- just returned an empty list. For a full day, it worked as shown.

You can call this same stuff through the REST API but I haven't tried that yet. Presumably the same values would work.

Upvotes: 9

Related Questions