Reputation: 12645
Suppose I have an S3 bucket named x.y.z
In this bucket, I have hundreds of files. But I only want to delete 2 files named purple.gif
and worksheet.xlsx
Can I do this from the AWS command line tool with a single call to rm
?
This did not work:
$ aws s3 rm s3://x.y.z/worksheet.xlsx s3://x.y.z/purple.gif
Unknown options: s3://x.y.z/purple.gif
From the manual, it doesn't seem like you can delete a list of files explicitly by name. Does anyone know a way to do it? I prefer not using the --recursive
flag.
Upvotes: 81
Views: 143345
Reputation: 175
This command deletes all files and directories in the bucket, leaving it completely empty:
aws s3 rm s3://bucketname --recursive
Upvotes: 14
Reputation: 126
FWIW: the below is an excerpt from my ~/.aws/cli/alias
hence relies upon dash
which is /bin/sh
on my Ubuntu (I wished it was bash
though):
myopts="--region us-east-1 --profile my-aws-cli-profile-name"
BUCKET_NAME="my-bucket-name"
for TYPE in Versions DeleteMarkers; do
if [ "$(aws $myopts s3api list-object-versions --bucket "$BUCKET_NAME" --max-items 1 --query "$TYPE" --output json)" != "null" ]; then
echo "INFO: Removing $TYPE"
aws $myopts s3api list-object-versions --bucket "$BUCKET_NAME" --query "$TYPE[].[Key, VersionId]" --output text | {
while read KEY VERSION; do
[ "$KEY" = "None" ] && continue
TARGET="$TARGET,{Key=$KEY,VersionId=$VERSION}"
TARGET=${TARGET#,}
COUNTER=$(expr $COUNTER + 1)
[ $(expr $COUNTER % 100) -ne 0 ] && continue
echo "INFO: $COUNTER"
aws $myopts s3api delete-objects --bucket "$BUCKET_NAME" --delete "Objects=[$TARGET],Quiet=true" >/dev/null || return $?
unset TARGET
done
if [ -n "$TARGET" ]; then
echo "INFO: $COUNTER"
aws $myopts s3api delete-objects --bucket "$BUCKET_NAME" --delete "Objects=[$TARGET],Quiet=true" >/dev/null || return $?
fi
}
fi
done
The 100
in the middle of the code snippet is how many objects to pass to each delete-objects
API call. This must not be greater than 1000
.
Upvotes: 0
Reputation: 1512
aws s3 rm s3://<bucketname>/2023/ --recursive --exclude '*' --include 'A*.csv'
None of the answers above mentions how to use a wildcard expression to choose multiple files and delete.
if your usecase is to choose multiple objects in S3 with a naming pattern and delete, the above command will be useful
Upvotes: 11
Reputation: 12869
You can delete multiple files using aws s3 rm
. If you want to delete all files in a specific folder, just use
aws s3 rm --recursive --region <AWS_REGION> s3://<AWS_BUCKET>/<FOLDER_PATH>/
first test it with the --dryrun
option!
Upvotes: 3
Reputation: 1
Quick way to delete a very large Folder in AWS
AWS_PROFILE=<AWS_PROFILE> AWS_BUCKET=<AWS_BUCKET> AWS_FOLDER=<AWS_FOLDER>; aws --profile $AWS_PROFILE s3 ls "s3://${AWS_BUCKET}/${AWS_FOLDER}/" | awk '{print $4}' | xargs -P8 -n1000 bash -c 'aws --profile '${AWS_PROFILE}' s3api delete-objects --bucket '${AWS_BUCKET}' --delete "Objects=[$(printf "{Key='${AWS_FOLDER}'/%s}," "$@")],Quiet=true" >/dev/null 2>&1'
PS: This might be launch 2/3 times because sometimes, some deletion fails...
Upvotes: 0
Reputation: 1315
Apparently aws s3 rm works only on individual files/objects.
Below is a bash command that constructs individual delete commands and then removes the objects one by one. Works with some success (might be bit slow, but works):
aws s3 ls s3://bucketname/foldername/ |
awk {'print "aws s3 rm s3://bucketname/foldername/" $4'} |
bash
The first two lines are meant to construct the "rm" commands and the 3rd line (bash) will execute them.
Note that you might face issues if your object names have spaces or funny characters. This is because "aws s3 ls" command won't list such objects (as of this writing)
Upvotes: 16
Reputation: 9847
s3 rm
cannot delete multiple files, but you can use s3api delete-objects
to achieve what you want here.
Example
aws s3api delete-objects --bucket x.y.z --delete '{"Objects":[{"Key":"worksheet.xlsx"},{"Key":"purple.gif"}]}'
Upvotes: 43
Reputation: 4429
I found this one useful through the command line. I had more than 4 million files and it took almost a week to empty the bucket. This comes handy as the AWS console is not descriptive with the logs.
Note: You need the jq
tool installed.
aws s3api list-object-versions --bucket YOURBUCKETNAMEHERE-processed \
--output json --query 'Versions[].[Key, VersionId]' \
| jq -r '.[] | "--key '\''" + .[0] + "'\'' --version-id " + .[1]' \
| xargs -L1 aws s3api delete-object --bucket YOURBUCKETNAMEHERE
Upvotes: 2
Reputation: 7831
If you are using AWS CLI you can filter LS results with grep regex and delete them. For example
aws s3 ls s3://BUCKET | awk '{print $4}' | grep -E -i '^2015-([0-9][0-9])\-([0-9][0-9])\-([0-9][0-9])\-([0-9][0-9])\-([0-9][0-9])\-([0-9a-zA-Z]*)' | xargs -I% bash -c 'aws s3 rm s3://BUCKET/%'
This is slow but it works
Upvotes: 2
Reputation: 77
This solution will work when you want to specify wildcard for object name.
aws s3 ls dmap-live-dwh-files/backup/mongodb/oms_api/hourly/ | grep order_2019_08_09_* | awk {'print "aws s3 rm s3://dmap-live-dwh-files/backup/mongodb/oms_api/hourly/" $4'} | bash
Upvotes: 1
Reputation: 10567
You can do this by providing an --exclude
or --include
argument multiple times. But, you'll have to use --recursive
for this to work.
When there are multiple filters, remember that the order of the filter parameters is important. The rule is the filters that appear later in the command take precedence over filters that appear earlier in the command.
aws s3 rm s3://x.y.z/ --recursive --exclude "*" --include "purple.gif" --include "worksheet.xlsx"
Here, all files will be excluded from the command except for purple.gif and worksheet.xlsx.
If you're unsure, always try a --dryrun
first and inspect which files will be deleted.
Source: Use of Exclude and Include Filters
Upvotes: 155