CODEWITHSUNDEEP
amazon-s3command-lineaws-cli
johnny
johnny

Reputation: 2122

AWS S3 CLI move files with a new name

I have millions of images that need to be renamed. All files have a 'C' in part of the filename that needs to be changed to a lower 'c'.

File name pattern:

How can I use AWS CLI with some kinda wild card to achieve this?

Source and destination buckets will be the same.

Upvotes: 3

Views: 6980

Answers (4)

q0rban
q0rban

Reputation: 987

Use the aws CLI with jq to parse the json output. Don't try to parse json with cut and tr as some of the other answers do.

Also, perform more strict matches. In the below example, we use the regular expression C\.jpg$ to match s3 items that only have a key ending in C.jpg.

And then be certain only to replace the trailing C.jpg by using the % substring replacement, which only replaces the trailing end of the string. Using ${var/C/c} as the other answers seem to suggest would change the first match of C in the entire string, which may not be what you desire. For example, with the provided filenames given in the question, ACD_112_2P-003-3C.jpg would become AcD_112_2P-003-3C.jpg using ${var/C/c}. Instead, use ${var/%C.jpg/c.jpg} which is sure to replace C.jpg at the end of the string with c.jpg.

s3_bucket=your-bucket-name
# select all items from the json that have a key that ends in C.jpg.
jq_cmd='.Contents[].Key | select(test("C\.jpg$"))'

for orig in $(aws s3api list-objects --bucket "$s3_bucket" | jq -r "$jq_cmd"); do
  # Replace the trailing C.jpg with c.jpg.
  new=${orig/%C.jpg/c.jpg}
  aws s3 mv "s3://$s3_bucket/$orig" "s3://$s3_bucket/$new"
done

Upvotes: 0

Ale M
Ale M

Reputation: 21

great it works! you are missing a final ) for the loop for

$ for f in $(aws s3api list-objects --bucket sample-bucket-1 --prefix "" --delimiter "" | grep replaceme | cut -d ":" -f 2 | tr -d , | tr -d \"); do aws s3 mv s3://sample-bucket-1/$f s3://sample-bucket-1/${f/replaceme/replaced}; done

$ for f in $(aws s3api list-objects --bucket sample-bucket-1 --prefix "" --delimiter "" | grep replaceme | cut -d ":" -f 2 | tr -d , | tr -d \"; do aws s3 mv s3://sample-bucket-1/$f s3://sample-bucket-1/${f/replaceme/replaced}; done

Upvotes: 2

phil
phil

Reputation: 518

Using aws cli and bash you can rename multiple files like so:

$ for f in $(aws s3api list-objects --bucket sample-bucket-1 --prefix "" --delimiter "" | grep replaceme | cut  -d ":" -f 2 | tr -d , | tr -d \";
do aws s3 mv s3://sample-bucket-1/$f s3://sample-bucket-1/${f/replaceme/replaced};
done

Where 'replaceme' is the part of the filename you wish to replace and 'replaced' is what you want to insert in the filename.

The following will extract the filename from the response:

 cut  -d ":" -f 2 | tr -d , | tr -d \"

If the files are within folders in a bucket place the folder name in prefix:

--prefix "folder_name/sub_folder_name"

Edited from here: http://gerardvivancos.com/2016/04/12/Single-and-Bulk-Renaming-of-Objects-in-Amazon-S3/

Upvotes: 4

John Rotenstein
John Rotenstein

Reputation: 269340

No, the AWS CLI cannot rename multiple files in one call.

Each rename operation will require a separate API call. It would probably be easier to do it with a program, such as a Python script. The script could retrieve a list of objects (or be given it), then make an Amazon S3 copy() call to copy the object to a new name, then delete the old object.

Upvotes: 2

Related Questions