How to list unused AWS S3 buckets and empty bucket using shell script?

Question

I am looking for list of unused s3 buckets from last 90 days and also for empty bucket list.

In order to get it, I have tried writing code as below:

#/bin/sh
for bucketlist in  $(aws s3api list-buckets --query "Buckets[].Name");
do
  listobjects=$(\
    aws s3api list-objects --bucket $bucketlist \
    --query 'Contents[?contains(LastModified, `2020-08-06`)]')
done

This code prints following output: [I have added results for only one bucket for reference]

{
    "Contents": [
        {
            "Key": "test2/image.png",
            "LastModified": "2020-08-06T17:19:10.000Z",
            "ETag": "\"xxxxxx\"",
            "Size": 179008,,
            "StorageClass": "STANDARD",
        }
    ]
}

Expectations:

1. In above code I want to print only bucket list which objects are not modified/used in last 90 days.
2. I am also looking for empty bucket list

I am not good in programming, Can anyone guide me on this? Thank you in advance for your support.

Answer 1

• To get the list of unused buckets, you can sort it by using last modified date which is sorting for 1 year or 2 years.
• The best way to sort the unused bucket is by identifying the bucket which is created manually but not by using cloudformation stack.
• below is the script that is used to list the s3 buckets which are created manually for individual use.

#!/bin/bash

echo "Running script"

# Define the regions to check
REGIONS=("us-east-1" "eu-west-1" "eu-west-2")

# Get the list of all S3 buckets
ALL_S3BUCKETS=$(aws s3 ls | cut -d' ' -f3)

# Remove duplicates from the list of all S3 buckets
ALL_S3BUCKETS=($(echo "${ALL_S3BUCKETS}" | tr ' ' '\n' | sort -u | tr '\n' ' '))

# Check each bucket in all regions
for BUCKET in "${ALL_S3BUCKETS[@]}"
do
    IS_ORPHANED=true
    for REGION in "${REGIONS[@]}"
    do
        ADD="--region $REGION"
        STACKS=$(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE --query 'StackSummaries[*].StackName' --output text $ADD)
        for STACK in $STACKS
        do
            STACK_BUCKETS=$(aws cloudformation list-stack-resources --stack-name $STACK --query "StackResourceSummaries[?ResourceType=='AWS::S3::Bucket'].PhysicalResourceId" --output text $ADD 2>> /dev/null)
            for STACK_BUCKET in $STACK_BUCKETS
            do
                if [[ "$STACK_BUCKET" == "$BUCKET" ]]
                then
                    IS_ORPHANED=false
                    break 2
                fi
            done
        done
    done

    if $IS_ORPHANED
    then
        echo "$BUCKET is ORPHANED in all regions"
    fi
done

Answer 2

I made this small bash script to find empty buckets in my account:

#!/bin/zsh
for b in $(aws s3 ls | cut -d" " -f3)
do
  echo -n $b
  if [[ "$(aws s3api list-objects-v2 --bucket $b --max-items 1)" == "" ]]
  then
    echo " BUCKET EMPTY"
  else
    echo ""
  fi
done

I listed the objects using the list-objects-v2 with maximum items of 1. If there are no items - the result is empty and I print "BUCKET EMPTY" alongside the bucket name.

• Note 1: You must have access to list the objects.
• Note 2: I'm not sure how it'll work for versioned buckets with deleted objects (appears to be empty, but actually contains older versions of deleted objects)

---

This works well but doesn't consider the non-current version objects. For that you can use "list-object-versions" then it will look for non-current version objects as well.

#!/bin/zsh for b in $(aws s3 ls | cut -d" " -f3) do echo -n $b if [[ "$(aws s3api list-object-versions --bucket $b --max-items 1)" == "" ]] then echo " <----- BUCKET EMPTY" else echo "" fi done

Answer 3

Here's a script I wrote today. It doesn't change anything, but it does give you the commandlines to make the changes.

#!/bin/bash
profile="default"
olddate="2020-01-01"
smallbucketsize=10

emptybucketlist=()
oldbucketlist=()
smallbucketlist=()

#for bucketlist in  $(aws s3api list-buckets  --profile $profile  | jq --raw-output '.Buckets[6,7,8,9].Name'); # test this script on just a few buckets
for bucketlist in  $(aws s3api list-buckets  --profile $profile  | jq --raw-output '.Buckets[].Name');
do
  echo "* $bucketlist"
  if [[ ! "$bucketlist" == *"shmr-logs" ]]; then
    listobjects=$(\
      aws s3api list-objects --bucket $bucketlist \
      --query 'Contents[*].Key' \
      --profile $profile)
#echo "==$listobjects=="
    if [[ "$listobjects" == "null" ]]; then
          echo "$bucketlist is empty"
          emptybucketlist+=("$bucketlist")
    else
      # get size
      aws s3 ls --summarize  --human-readable --recursive --profile $profile s3://$bucketlist | tail -n1

      # get number of files
      filecount=$(echo $listobjects | jq length )
      echo "contains $filecount files"
      if [[ $filecount -lt $smallbucketsize ]]; then
          smallbucketlist+=("$bucketlist")
      fi

      # get number of files older than $olddate
      listoldobjects=$(\
        aws s3api list-objects --bucket $bucketlist \
        --query "Contents[?LastModified<=\`$olddate\`]" \
        --profile $profile)
      oldfilecount=$(echo $listoldobjects | jq length )
      echo "contains $oldfilecount old files"

      # check if all files are old
      if [[ $filecount -eq $oldfilecount ]]; then
        echo "all the files are old"
        oldbucketlist+=("$bucketlist")

      fi
    fi
  fi
done
echo -e "\n\n"

echo "check the contents of these buckets which only contain old files"
for oldbuckets in ${oldbucketlist[@]};
do
  echo "$oldbuckets"
done
echo -e "\n\n"

echo "check the contents of these buckets which don't have many files"
for smallbuckets in ${smallbucketlist[@]};
do
  echo "aws s3api list-objects --bucket $smallbuckets --query 'Contents[*].Key' --profile $profile"
done
echo -e "\n\n"

echo "consider deleting these empty buckets"
for emptybuckets in "${emptybucketlist[@]}";
do
  echo "aws s3api delete-bucket --profile $profile --bucket $emptybuckets"
done