Jiya
Jiya

Reputation: 243

How to list unused AWS S3 buckets and empty bucket using shell script?

I am looking for list of unused s3 buckets from last 90 days and also for empty bucket list.

In order to get it, I have tried writing code as below:

#/bin/sh
for bucketlist in  $(aws s3api list-buckets --query "Buckets[].Name");
do
  listobjects=$(\
    aws s3api list-objects --bucket $bucketlist \
    --query 'Contents[?contains(LastModified, `2020-08-06`)]')
done

This code prints following output: [I have added results for only one bucket for reference]

{
    "Contents": [
        {
            "Key": "test2/image.png",
            "LastModified": "2020-08-06T17:19:10.000Z",
            "ETag": "\"xxxxxx\"",
            "Size": 179008,,
            "StorageClass": "STANDARD",
        }
    ]
}

Expectations:

  1. In above code I want to print only bucket list which objects are not modified/used in last 90 days.
  2. I am also looking for empty bucket list

I am not good in programming, Can anyone guide me on this? Thank you in advance for your support.

Upvotes: 2

Views: 8865

Answers (3)

Roja Reddy
Roja Reddy

Reputation: 9

  • To get the list of unused buckets, you can sort it by using last modified date which is sorting for 1 year or 2 years.
  • The best way to sort the unused bucket is by identifying the bucket which is created manually but not by using cloudformation stack.
  • below is the script that is used to list the s3 buckets which are created manually for individual use.
#!/bin/bash

echo "Running script"

# Define the regions to check
REGIONS=("us-east-1" "eu-west-1" "eu-west-2")

# Get the list of all S3 buckets
ALL_S3BUCKETS=$(aws s3 ls | cut -d' ' -f3)

# Remove duplicates from the list of all S3 buckets
ALL_S3BUCKETS=($(echo "${ALL_S3BUCKETS}" | tr ' ' '\n' | sort -u | tr '\n' ' '))

# Check each bucket in all regions
for BUCKET in "${ALL_S3BUCKETS[@]}"
do
    IS_ORPHANED=true
    for REGION in "${REGIONS[@]}"
    do
        ADD="--region $REGION"
        STACKS=$(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE --query 'StackSummaries[*].StackName' --output text $ADD)
        for STACK in $STACKS
        do
            STACK_BUCKETS=$(aws cloudformation list-stack-resources --stack-name $STACK --query "StackResourceSummaries[?ResourceType=='AWS::S3::Bucket'].PhysicalResourceId" --output text $ADD 2>> /dev/null)
            for STACK_BUCKET in $STACK_BUCKETS
            do
                if [[ "$STACK_BUCKET" == "$BUCKET" ]]
                then
                    IS_ORPHANED=false
                    break 2
                fi
            done
        done
    done

    if $IS_ORPHANED
    then
        echo "$BUCKET is ORPHANED in all regions"
    fi
done

Upvotes: 0

Uziel Sulkies
Uziel Sulkies

Reputation: 628

I made this small bash script to find empty buckets in my account:

#!/bin/zsh
for b in $(aws s3 ls | cut -d" " -f3)
do
  echo -n $b
  if [[ "$(aws s3api list-objects-v2 --bucket $b --max-items 1)" == "" ]]
  then
    echo " BUCKET EMPTY"
  else
    echo ""
  fi
done

I listed the objects using the list-objects-v2 with maximum items of 1. If there are no items - the result is empty and I print "BUCKET EMPTY" alongside the bucket name.

  • Note 1: You must have access to list the objects.
  • Note 2: I'm not sure how it'll work for versioned buckets with deleted objects (appears to be empty, but actually contains older versions of deleted objects)

This works well but doesn't consider the non-current version objects. For that you can use "list-object-versions" then it will look for non-current version objects as well.

#!/bin/zsh for b in $(aws s3 ls | cut -d" " -f3) do echo -n $b if [[ "$(aws s3api list-object-versions --bucket $b --max-items 1)" == "" ]] then echo " <----- BUCKET EMPTY" else echo "" fi done

Upvotes: 5

andrew lorien
andrew lorien

Reputation: 2688

Here's a script I wrote today. It doesn't change anything, but it does give you the commandlines to make the changes.

#!/bin/bash
profile="default"
olddate="2020-01-01"
smallbucketsize=10

emptybucketlist=()
oldbucketlist=()
smallbucketlist=()

#for bucketlist in  $(aws s3api list-buckets  --profile $profile  | jq --raw-output '.Buckets[6,7,8,9].Name'); # test this script on just a few buckets
for bucketlist in  $(aws s3api list-buckets  --profile $profile  | jq --raw-output '.Buckets[].Name');
do
  echo "* $bucketlist"
  if [[ ! "$bucketlist" == *"shmr-logs" ]]; then
    listobjects=$(\
      aws s3api list-objects --bucket $bucketlist \
      --query 'Contents[*].Key' \
      --profile $profile)
#echo "==$listobjects=="
    if [[ "$listobjects" == "null" ]]; then
          echo "$bucketlist is empty"
          emptybucketlist+=("$bucketlist")
    else
      # get size
      aws s3 ls --summarize  --human-readable --recursive --profile $profile s3://$bucketlist | tail -n1

      # get number of files
      filecount=$(echo $listobjects | jq length )
      echo "contains $filecount files"
      if [[ $filecount -lt $smallbucketsize ]]; then
          smallbucketlist+=("$bucketlist")
      fi

      # get number of files older than $olddate
      listoldobjects=$(\
        aws s3api list-objects --bucket $bucketlist \
        --query "Contents[?LastModified<=\`$olddate\`]" \
        --profile $profile)
      oldfilecount=$(echo $listoldobjects | jq length )
      echo "contains $oldfilecount old files"

      # check if all files are old
      if [[ $filecount -eq $oldfilecount ]]; then
        echo "all the files are old"
        oldbucketlist+=("$bucketlist")

      fi
    fi
  fi
done
echo -e "\n\n"

echo "check the contents of these buckets which only contain old files"
for oldbuckets in ${oldbucketlist[@]};
do
  echo "$oldbuckets"
done
echo -e "\n\n"

echo "check the contents of these buckets which don't have many files"
for smallbuckets in ${smallbucketlist[@]};
do
  echo "aws s3api list-objects --bucket $smallbuckets --query 'Contents[*].Key' --profile $profile"
done
echo -e "\n\n"

echo "consider deleting these empty buckets"
for emptybuckets in "${emptybucketlist[@]}";
do
  echo "aws s3api delete-bucket --profile $profile --bucket $emptybuckets"
done

Upvotes: 3

Related Questions