CODEWITHSUNDEEP
amazon-web-servicesamazon-s3command-line-interfaceaws-clis3cmd
NobleUplift
NobleUplift

Reputation: 6024

How do I delete a versioned bucket in AWS S3 using the CLI?

I have tried both s3cmd:

$ s3cmd -r -f -v del s3://my-versioned-bucket/

And the AWS CLI:

$ aws s3 rm s3://my-versioned-bucket/ --recursive

But both of these commands simply add DELETE markers to S3. The command for removing a bucket also doesn't work (from the AWS CLI):

$ aws s3 rb s3://my-versioned-bucket/ --force
Cleaning up. Please wait...
Completed 1 part(s) with ... file(s) remaining
remove_bucket failed: s3://my-versioned-bucket/ A client error (BucketNotEmpty) occurred when calling the DeleteBucket operation: The bucket you tried to delete is not empty. You must delete all versions in the bucket.

Ok... how? There's no information in their documentation for this. S3Cmd says it's a 'fully-featured' S3 command-line tool, but it makes no reference to versions other than its own. Is there any way to do this without using the web interface, which will take forever and requires me to keep my laptop on?

Upvotes: 142

Views: 134629

Answers (28)

Chiranjeevi Ganesh
Chiranjeevi Ganesh

Reputation: 1

Below script will help you delete huge data in S3 bucket with object versioning enabled and delete markers

#!/bin/bash

export AWS_PROFILE=ABC

bucket_name="XYZ"
max_items_per_page=1000  # Adjust this as needed

set -e

echo "Removing all versions from $bucket_name"

next_token=""

while true; do
    result=$(aws s3api list-object-versions --bucket "$bucket_name" --max-items $max_items_per_page --starting-token "$next_token")
    versions=$(echo "$result" | jq '.Versions')
    markers=$(echo "$result" | jq '.DeleteMarkers')

    if [ "$versions" == "null" ] && [ "$markers" == "null" ]; then
        echo "No more versions to delete."
        break
    fi

    # Removing files
    for version in $(echo "$versions" | jq -r '.[] | @base64'); do
        version=$(echo "$version" | base64 --decode)

        key=$(echo "$version" | jq -r .Key)
        versionId=$(echo "$version" | jq -r .VersionId)
        cmd="aws s3api delete-object --bucket $bucket_name --key $key --version-id $versionId"
        echo "$cmd"
        $cmd
    done

    # Removing delete markers
    for marker in $(echo "$markers" | jq -r '.[] | @base64'); do
        marker=$(echo "$marker" | base64 --decode)

        key=$(echo "$marker" | jq -r .Key)
        versionId=$(echo "$marker" | jq -r .VersionId)
        cmd="aws s3api delete-object --bucket $bucket_name --key $key --version-id $versionId"
        echo "$cmd"
        $cmd
    done

    next_token=$(echo "$result" | jq -r '.NextToken')
done

Upvotes: 0

holi0317
holi0317

Reputation: 11

If you got rclone installed, you can use rclone purge command to delete versioned bucket.

Follow the configuration guide and setup s3 remote, following snippet uses s3 as the name when configuring the remote.

rclone purge --verbose s3:my-versioned-bucket

Upvotes: 0

Alexandre Hamon
Alexandre Hamon

Reputation: 1402

This two bash lines are enough for me to enable the bucket deletion !

1: Delete objects aws s3api delete-objects --bucket ${buckettoempty} --delete "$(aws s3api list-object-versions --bucket ${buckettoempty} --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

2: Delete markers aws s3api delete-objects --bucket ${buckettoempty} --delete "$(aws s3api list-object-versions --bucket ${buckettoempty} --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"

EDIT: If you have 500+ items in your S3 bucket and you get Argument list too long error, you will need to loop on the above command to empty your bucket :

NB_OBJECTS=$(aws s3api list-object-versions --bucket ${buckettoempty} --query='length(Versions[*] || `[]` )' | awk '{ print $1 }')
echo "      '${NB_OBJECTS}' objects to remove"
if [[ "$NB_OBJECTS" != "0" ]]; then
  start=$SECONDS
  while [[ $NB_OBJECTS -gt 0 ]]
  do
    aws s3api delete-objects --bucket ${buckettoempty} --delete "$(aws s3api list-object-versions --bucket ${buckettoempty} --max-items 500 --query='{Objects: Versions[0:500].{Key:Key,VersionId:VersionId}}')" --query 'length(Deleted[*] || `[]` )' > /dev/null
    NB_OBJECTS=$((NB_OBJECTS  > 500 ? NB_OBJECTS - 500 : 0))
    echo "      Removed batch of Objects... Remaining : $NB_OBJECTS ($(( SECONDS - start ))s)"
  done
fi

NB_OBJECTS=$(aws s3api list-object-versions --bucket ${buckettoempty} --query='length(DeleteMarkers[*] || `[]` )' | awk '{ print $1 }')
echo "      '${NB_OBJECTS}' markers to remove"
if [[ "$NB_OBJECTS" != "0" ]]; then
  start=$SECONDS
  while [[ $NB_OBJECTS -gt 0 ]]
  do
    aws s3api delete-objects --bucket ${buckettoempty} --delete "$(aws s3api list-object-versions --bucket ${buckettoempty} --max-items 500 --query='{Objects: DeleteMarkers[0:500].{Key:Key,VersionId:VersionId}}')" --query 'length(Deleted[*] || `[]` )' > /dev/null
    NB_OBJECTS=$((NB_OBJECTS  > 500 ? NB_OBJECTS - 500 : 0))
    echo "      Removed batch of Markers... Remaining : $NB_OBJECTS (took $(( SECONDS - start ))s)"
  done
fi

Upvotes: 47

Abe Voelker
Abe Voelker

Reputation: 31622

I ran into the same limitation of the AWS CLI. I found the easiest solution to be to use Python and boto3:

#!/usr/bin/env python

BUCKET = 'your-bucket-here'

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket(BUCKET)
bucket.object_versions.delete()

# if you want to delete the now-empty bucket as well, uncomment this line:
#bucket.delete()

A previous version of this answer used boto but that solution had performance issues with large numbers of keys as Chuckles pointed out.

Upvotes: 174

oleksii
oleksii

Reputation: 35925

I needed to delete older object versions but keep the current version in the bucket. Code uses iterators, works on buckets of any size with any number of objects.

import boto3
from itertools import islice

bucket = boto3.resource('s3').Bucket('bucket_name'))
all_versions = bucket.object_versions.all()
stale_versions = iter(filter(lambda x: not x.is_latest, all_versions))
pages = iter(lambda: tuple(islice(stale_versions, 1000)), ())
for page in pages:
    bucket.delete_objects(
        Delete={
            'Objects': [{
                'Key': item.key,
                'VersionId': item.version_id
            } for item in page]
        })

Upvotes: 0

Dunedan
Dunedan

Reputation: 8455

Using boto3 it's even easier than with the proposed boto solution to delete all object versions in an S3 bucket:

#!/usr/bin/env python
import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('your-bucket-name')
bucket.object_versions.all().delete()

Works fine also for very large amounts of object versions, although it might take some time in that case.

Upvotes: 60

TryTryAgain
TryTryAgain

Reputation: 7830

In the same vein as https://stackoverflow.com/a/63613510/805031 ... this is what I use to clean up accounts before closing them:

# If the data is too large, apply LCP to remove all objects within a day

# Create lifecycle-expire.json with the LCP required to purge all objects
# Based on instructions from: https://aws.amazon.com/premiumsupport/knowledge-center/s3-empty-bucket-lifecycle-rule/
cat << JSON > lifecycle-expire.json
{
    "Rules": [
        {
            "ID": "remove-all-objects-asap",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "Expiration": {
                "Days": 1
            },
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 1
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        },
        {
            "ID": "remove-expired-delete-markers",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            }
        }
    ]
}
JSON

# Apply to ALL buckets
aws s3 ls | cut -d" " -f 3 | xargs -I{} aws s3api put-bucket-lifecycle-configuration --bucket {} --lifecycle-configuration file://lifecycle-expire.json

# Apply to a single bucket; replace $BUCKET_NAME
aws s3api put-bucket-lifecycle-configuration --bucket $BUCKET_NAME --lifecycle-configuration file://lifecycle-expire.json

...then a day later you can come back and delete the buckets using something like:

# To force empty/delete all buckets
aws s3 ls | cut -d" " -f 3 | xargs -I{} aws s3 rb s3://{} --force

# To remove only empty buckets
aws s3 ls | cut -d" " -f 3 | xargs -I{} aws s3 rb s3://{}

# To force empty/delete a single bucket; replace $BUCKET_NAME
aws s3 rb s3://$BUCKET_NAME --force

It saves a lot of time and money so worth doing when you have many TBs to delete.

Upvotes: 1

Andron
Andron

Reputation: 6631

For my case, I wanted to be sure that all objects for specific prefixes would be deleted. So, we generate a list of all objects for each prefix, divide it by 1k records (AWS limitation), and delete them.

Please note that AWS CLI and jq must be installed and configured.

A text file with prefixes that we want to delete was created (in the example below prefixes.txt).

The format is:

prefix1
prefix2

And this is a shell script (also please change the BUCKET_NAME with the real name):

#!/bin/sh

BUCKET="BUCKET_NAME"
PREFIXES_FILE="prefixes.txt"

if [ -f "$PREFIXES_FILE" ]; then
    while read -r current_prefix
    do
        printf '***** PREFIX %s *****\n' "$current_prefix"

        OLD_OBJECTS_FILE="$current_prefix-all.json"

        if [ -f "$OLD_OBJECTS_FILE" ]; then
            printf 'Deleted %s...\n' "$OLD_OBJECTS_FILE"

            rm "$OLD_OBJECTS_FILE"
        fi

        cmd="aws s3api list-object-versions --bucket \"$BUCKET\" --prefix \"$current_prefix/\" --query \"[Versions,DeleteMarkers][].{Key: Key, VersionId: VersionId}\"  >> $OLD_OBJECTS_FILE"
        echo "$cmd"
        eval "$cmd"

        no_of_obj=$(cat "$OLD_OBJECTS_FILE" | jq 'length')
        i=0
        page=0

        #Get old version Objects
        echo "Objects versions count: $no_of_obj"

        while [ $i -lt "$no_of_obj" ]
        do
            next=$((i+999))
            old_versions=$(cat "$OLD_OBJECTS_FILE" | jq '.[] | {Key,VersionId}' | jq -s '.' | jq .[$i:$next])
            paged_file_name="$current_prefix-page-$page.json"
            cat << EOF > "$paged_file_name"
{"Objects":$old_versions, "Quiet":true}
EOF
            echo "Deleting records from $i - $next"
            cmd="aws s3api delete-objects --bucket \"$BUCKET\" --delete file://$paged_file_name"
            echo "$cmd"
            eval "$cmd"

            i=$((i+1000))
            page=$((page+1))
        done

    done < "$PREFIXES_FILE"
else
    echo "$PREFIXES_FILE does not exist."
fi

If you want just to check the list of objects and don't delete them immediately - please comment/remove the last eval "$cmd".

Upvotes: 0

deniss
deniss

Reputation: 1

If you use AWS SDK for JavaScript S3 Client for Node.js (@aws-sdk/client-s3), you can use following code:

const { S3Client, ListObjectsCommand } = require('@aws-sdk/client-s3')


const endpoint = 'YOUR_END_POINT'
const region = 'YOUR_REGION'

// Create an Amazon S3 service client object.
const s3Client = new S3Client({ region, endpoint })

const deleteEverythingInBucket = async bucketName => {
  console.log('Deleting all object in the bucket')

  const bucketParams = {
    Bucket: bucketName
  }

  try {
    const command = new ListObjectsCommand(bucketParams)
    const data = await s3Client.send(command)
    console.log('Bucket Data', JSON.stringify(data))
    if (data?.Contents?.length > 0) {
      console.log('Removing objects in the bucket', data.Contents.length)
      for (const object of data.Contents) {
        console.log('Removing object', object)
        if (object.Key) {
          try {
            await deleteFromS3({
              Bucket: bucketName,
              Key: object.Key
            })
          } catch (err) {
            console.log('Error on object delete', err)
          }
        }
      }
    }
  } catch (err) {
    console.log('Error creating presigned URL', err)
  }
}

Upvotes: 0

Volkan Paksoy
Volkan Paksoy

Reputation: 6977

Even though technically it's not AWS CLI, I'd recommend using AWS Tools for Powershell for this task. Then you can use the simple command as below:

Remove-S3Bucket -BucketName {bucket-name} -DeleteBucketContent -Force -Region {region}

As stated in the documentation, DeleteBucketContent flag does the following:

"If set, all remaining objects and/or object versions in the bucket are deleted proir (sic) to the bucket itself being deleted"

Reference: https://docs.aws.amazon.com/powershell/latest/reference/items/Remove-S3Bucket.html

Upvotes: 7

Andrew Chong
Andrew Chong

Reputation: 174

I improved the boto3 answer with Python3 and argv.

  1. Save the following script as something like s3_rm.py.
#!/usr/bin/env python3
import sys
import boto3

def main():
    args = sys.argv[1:]
    if (len(args) < 1):
        print("Usage: {} s3_bucket_name".format(sys.argv[0]))
        exit()

    s3 = boto3.resource('s3')
    bucket = s3.Bucket(args[0])
    bucket.object_versions.delete()

    # if you want to delete the now-empty bucket as well, uncomment this line:
    #bucket.delete()

if __name__ == "__main__": 
    main()
  1. Add chmod +x s3_rm.py.
  2. Run the function like ./s3_rm.py my_bucket_name.

Upvotes: 1

JBernales
JBernales

Reputation: 149

Simple bash loop I've found and implemented for N buckets:

for b in $(ListOfBuckets); do \
    echo "Emptying $b"; \
    aws s3api delete-objects --bucket $b --delete "$(aws s3api list-object-versions --bucket $b --output=json --query='{Objects: *[].{Key:Key,VersionId:VersionId}}')"; \
done

Upvotes: 10

Zakaria
Zakaria

Reputation: 863

You can use aws-cli to delete s3 bucket

aws s3 rb s3://your-bucket-name

If aws cli is not installed in your computer you can your following commands: For Linux or ubuntu:

sudo apt-get install aws-cli

Then check it is installed or not by:

aws --version

Now configure it by providing aws-access-credentials

aws configure

Then give the access key and secret access key and your region

Upvotes: -2

Alexander Pogrebnyak
Alexander Pogrebnyak

Reputation: 45596

Looks like as of now, there is an Empty button in the AWS S3 console. Empty Button

Just select your bucket and click on it. It will ask you to confirm your decision by typing permanently delete Note, this will not delete the bucket itself.

Upvotes: 35

Bob Ritchie
Bob Ritchie

Reputation: 101

If you want pure CLI approach (with jq):

aws s3api list-object-versions \
          --bucket $bucket \
          --region $region \
          --query "Versions[].Key"  \
          --output json | jq 'unique' | jq -r '.[]' | while read key; do
   echo "deleting versions of $key"
   aws s3api list-object-versions \
          --bucket $bucket \
          --region $region \
          --prefix $key \
          --query "Versions[].VersionId"  \
          --output json | jq 'unique' | jq -r '.[]' | while read version; do
     echo "deleting $version"
     aws s3api delete-object \
          --bucket $bucket \
          --key $key \
          --version-id $version \
          --region $region
   done
done

Upvotes: 10

muhqu
muhqu

Reputation: 12809

If you have to delete/empty large S3 buckets, it becomes quite inefficient (and expensive) to delete every single object and version. It's often more convenient to let AWS expire all objects and versions.

aws s3api put-bucket-lifecycle-configuration \
  --lifecycle-configuration '{"Rules":[{
      "ID":"empty-bucket",
      "Status":"Enabled",
      "Prefix":"",
      "Expiration":{"Days":1},
      "NoncurrentVersionExpiration":{"NoncurrentDays":1}
    }]}' \
  --bucket YOUR-BUCKET

Then you just have to wait 1 day and the bucket can be deleted with:

aws s3api delete-bucket --bucket YOUR-BUCKET

Upvotes: 19

user2586441
user2586441

Reputation: 131

To add to python solutions provided here: if you are getting boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request error, try creating ~/.boto file with the following data:

[Credentials]
aws_access_key_id = aws_access_key_id
aws_secret_access_key = aws_secret_access_key
[s3]
host=s3.eu-central-1.amazonaws.com
aws_access_key_id = aws_access_key_id
aws_secret_access_key = aws_secret_access_key

Helped me to delete bucket in Frankfurt region.

Original answer: https://stackoverflow.com/a/41200567/2586441

Upvotes: 0

ojhurst
ojhurst

Reputation: 412

You can do this from the AWS Console using Lifecycle Rules.

Open the bucket in question. Click the Management tab at the top. Make sure the Lifecycle Sub Tab is selected. Click + Add lifecycle rule

On Step 1 (Name and scope) enter a rule name (e.g. removeall) Click Next to Step 2 (Transitions) Leave this as is and click Next.

You are now on the 3. Expiration step. Check the checkboxes for both Current Version and Previous Versions. Click the checkbox for "Expire current version of object" and enter the number 1 for "After _____ days from object creation Click the checkbox for "Permanently delete previous versions" and enter the number 1 for "After _____ days from becoming a previous version"

click the checkbox for "Clean up incomplete multipart uploads" and enter the number 1 for "After ____ days from start of upload" Click Next Review what you just did.
Click Save

Come back in a day and see how it is doing.

enter image description here

Upvotes: 2

gotit
gotit

Reputation: 727

For those using multiple profiles via ~/.aws/config

import boto3

PROFILE = "my_profile"
BUCKET = "my_bucket"

session = boto3.Session(profile_name = PROFILE)
s3 = session.resource('s3')
bucket = s3.Bucket(BUCKET)
bucket.object_versions.delete()

Upvotes: 13

roady
roady

Reputation: 597

This works for me. Maybe running later versions of something and above > 1000 items. been running a couple of million files now. However its still not finished after half a day and no means to validate in AWS GUI =/

# Set bucket name to clearout
BUCKET = 'bucket-to-clear'

import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket(BUCKET)

max_len         = 1000      # max 1000 items at one req
chunk_counter   = 0         # just to keep track
keys            = []        # collect to delete

# clear files
def clearout():
    global bucket
    global chunk_counter
    global keys
    result = bucket.delete_objects(Delete=dict(Objects=keys))

    if result["ResponseMetadata"]["HTTPStatusCode"] != 200:
        print("Issue with response")
        print(result)

    chunk_counter += 1
    keys = []
    print(". {n} chunks so far".format(n=chunk_counter))
    return

# start
for key in bucket.object_versions.all():
    item = {'Key': key.object_key, 'VersionId': key.id}
    keys.append(item)
    if len(keys) >= max_len:
        clearout()

# make sure last files are cleared as well
if len(keys) > 0:
    clearout()

print("")
print("Done, {n} items deleted".format(n=chunk_counter*max_len))
#bucket.delete() #as per usual uncomment if you're sure!

Upvotes: 0

Nitin
Nitin

Reputation: 11236

Here is a one liner you can just cut and paste into the command line to delete all versions and delete markers (it requires aws tools, replace yourbucket-name-backup with your bucket name)

echo '#!/bin/bash' > deleteBucketScript.sh \
&& aws --output text s3api list-object-versions --bucket $BUCKET_TO_PERGE \
| grep -E "^VERSIONS" |\
awk '{print "aws s3api delete-object --bucket $BUCKET_TO_PERGE --key "$4" --version-id "$8";"}' >> \
deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh; echo '#!/bin/bash' > \
deleteBucketScript.sh && aws --output text s3api list-object-versions --bucket $BUCKET_TO_PERGE \
| grep -E "^DELETEMARKERS" | grep -v "null" \
| awk '{print "aws s3api delete-object --bucket $BUCKET_TO_PERGE --key "$3" --version-id "$5";"}' >> \
deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh;

then you could use:

aws s3 rb s3://bucket-name --force

Upvotes: 15

user2402169
user2402169

Reputation: 1

https://gist.github.com/wknapik/191619bfa650b8572115cd07197f3baf

#!/usr/bin/env bash

set -eEo pipefail
shopt -s inherit_errexit >/dev/null 2>&1 || true

if [[ ! "$#" -eq 2 || "$1" != --bucket ]]; then
    echo -e "USAGE: $(basename "$0") --bucket <bucket>"
    exit 2
fi

# $@ := bucket_name
empty_bucket() {
    local -r bucket="${1:?}"
    for object_type in Versions DeleteMarkers; do
        local opt=() next_token=""
        while [[ "$next_token" != null ]]; do
            page="$(aws s3api list-object-versions --bucket "$bucket" --output json --max-items 1000 "${opt[@]}" \
                        --query="[{Objects: ${object_type}[].{Key:Key, VersionId:VersionId}}, NextToken]")"
            objects="$(jq -r '.[0]' <<<"$page")"
            next_token="$(jq -r '.[1]' <<<"$page")"
            case "$(jq -r .Objects <<<"$objects")" in
                '[]'|null) break;;
                *) opt=(--starting-token "$next_token")
                   aws s3api delete-objects --bucket "$bucket" --delete "$objects";;
            esac
        done
    done
}

empty_bucket "${2#s3://}"

E.g. empty_bucket.sh --bucket foo

This will delete all object versions and delete markers in a bucket in batches of 1000. Afterwards, the bucket can be deleted with aws s3 rb s3://foo.

Requires bash, awscli and jq.

Upvotes: 0

slm
slm

Reputation: 16436

By far the easiest method I've found is to use this CLI tool, s3wipe. It's provided as a docker container so you can use it like so:

$ docker run -it --rm slmingol/s3wipe --help
usage: s3wipe [-h] --path PATH [--id ID] [--key KEY] [--dryrun] [--quiet]
              [--batchsize BATCHSIZE] [--maxqueue MAXQUEUE]
              [--maxthreads MAXTHREADS] [--delbucket] [--region REGION]

Recursively delete all keys in an S3 path

optional arguments:
  -h, --help               show this help message and exit
  --path PATH              S3 path to delete (e.g. s3://bucket/path)
  --id ID                  Your AWS access key ID
  --key KEY                Your AWS secret access key
  --dryrun                 Don't delete. Print what we would have deleted
  --quiet                  Suprress all non-error output
  --batchsize BATCHSIZE    # of keys to batch delete (default 100)
  --maxqueue MAXQUEUE      Max size of deletion queue (default 10k)
  --maxthreads MAXTHREADS  Max number of threads (default 100)
  --delbucket              If S3 path is a bucket path, delete the bucket also
  --region REGION          Region of target S3 bucket. Default vaue `us-
                           east-1`

Example

Here's an example where I'm deleting all the versioned objects in a bucket and then deleting the bucket:

$ docker run -it --rm slmingol/s3wipe \
   --id $(aws configure get default.aws_access_key_id) \
   --key $(aws configure get default.aws_secret_access_key) \
   --path s3://bw-tf-backends-aws-example-logs \
   --delbucket
[2019-02-20@03:39:16] INFO: Deleting from bucket: bw-tf-backends-aws-example-logs, path: None
[2019-02-20@03:39:16] INFO: Getting subdirs to feed to list threads
[2019-02-20@03:39:18] INFO: Done deleting keys
[2019-02-20@03:39:18] INFO: Bucket is empty.  Attempting to remove bucket

How it works

There's a bit to unpack here but the above is doing the following:

  • docker run -it --rm mikelorant/s3wipe - runs s3wipe container interactively and deletes it after each execution
  • --id & --key - passing our access key and access id in
  • aws configure get default.aws_access_key_id - retrieves our key id
  • aws configure get default.aws_secret_access_key - retrieves our key secret
  • --path s3://bw-tf-backends-aws-example-logs - bucket that we want to delete
  • --delbucket - deletes bucket once emptied

References

Upvotes: 0

Cheers
Cheers

Reputation: 639

You can delete all the objects in the versioned s3 bucket. But I don't know how to delete specific objects.

$ aws s3api delete-objects \
      --bucket <value> \
      --delete "$(aws s3api list-object-versions \
      --bucket <value> | \
      jq '{Objects: [.Versions[] | {Key:.Key, VersionId : .VersionId}], Quiet: false}')"

Alternatively without jq:

$ aws s3api delete-objects \
    --bucket ${bucket_name} \
    --delete "$(aws s3api list-object-versions \
    --bucket "${bucket_name}" \
    --output=json \
    --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

Upvotes: 59

Oliver
Oliver

Reputation: 29591

I found the other answers either incomplete or requiring external dependencies to be installed (like boto), so here is one that is inspired by those but goes a little deeper.

As documented in Working with Delete Markers, before a versioned bucket can be removed, all its versions must be completely deleted, which is a 2-step process:

  1. "delete" all version objects in the bucket, which marks them as deleted but does not actually delete them
  2. complete the deletion by deleting all the deletion marker objects

Here is the pure CLI solution that worked for me (inspired by the other answers): 

#!/usr/bin/env bash

bucket_name=...

del_s3_bucket_obj()
{
    local bucket_name=$1
    local obj_type=$2
    local query="{Objects: $obj_type[].{Key:Key,VersionId:VersionId}}"
    local s3_objects=$(aws s3api list-object-versions --bucket ${bucket_name} --output=json --query="$query")
    if ! (echo $s3_objects | grep -q '"Objects": null'); then
        aws s3api delete-objects --bucket "${bucket_name}" --delete "$s3_objects"
    fi
}

del_s3_bucket_obj ${bucket_name} 'Versions'
del_s3_bucket_obj ${bucket_name} 'DeleteMarkers'

Once this is done, the following will work: 

aws s3 rb "s3://${bucket_name}"

Not sure how it will fare with 1000+ objects though, if anyone can report that would be awesome.

Upvotes: 0

Tiger peng
Tiger peng

Reputation: 747

  1. For deleting specify object(s), using jq filter.
  2. You may need cleanup the 'DeleteMarkers' not just 'Versions'.
  3. Using $() instead of ``, you may embed variables for bucket-name and key-value.
aws s3api delete-objects --bucket bucket-name --delete "$(aws s3api list-object-versions --bucket bucket-name | jq -M '{Objects: [.["Versions","DeleteMarkers"][]|select(.Key == "key-value")| {Key:.Key, VersionId : .VersionId}], Quiet: false}')"

Upvotes: 7

chuckwired
chuckwired

Reputation: 147

I ran into issues with Abe's solution as the list_buckets generator is used to create a massive list called all_keys and I spent an hour without it ever completing. This tweak seems to work better for me, I had close to a million objects in my bucket and counting!

import boto

s3 = boto.connect_s3()
bucket = s3.get_bucket("your-bucket-name-here")

chunk_counter = 0 #this is simply a nice to have
keys = []
for key in bucket.list_versions():
    keys.append(key)
    if len(keys) > 1000:
        bucket.delete_keys(keys)
        chunk_counter += 1
        keys = []
        print("Another 1000 done.... {n} chunks so far".format(n=chunk_counter))

#bucket.delete() #as per usual uncomment if you're sure!

Hopefully this helps anyone else encountering this S3 nightmare!

Upvotes: 7

Julio Faerman
Julio Faerman

Reputation: 13501

One way to do it is iterate through the versions and delete them. A bit tricky on the CLI, but as you mentioned Java, that would be more straightforward:

AmazonS3Client s3 = new AmazonS3Client();
String bucketName = "deleteversions-"+UUID.randomUUID();

//Creates Bucket
s3.createBucket(bucketName);

//Enable Versioning
BucketVersioningConfiguration configuration = new BucketVersioningConfiguration(ENABLED);
s3.setBucketVersioningConfiguration(new SetBucketVersioningConfigurationRequest(bucketName, configuration ));

//Puts versions
s3.putObject(bucketName, "some-key",new ByteArrayInputStream("some-bytes".getBytes()), null);
s3.putObject(bucketName, "some-key",new ByteArrayInputStream("other-bytes".getBytes()), null);

//Removes all versions
for ( S3VersionSummary version : S3Versions.inBucket(s3, bucketName) ) {
    String key = version.getKey();
    String versionId = version.getVersionId();          
    s3.deleteVersion(bucketName, key, versionId);
}

//Removes the bucket
s3.deleteBucket(bucketName);
System.out.println("Done!");

You can also batch delete calls for efficiency if needed.

Upvotes: 11

Related Questions