CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-cli
K.Tom
K.Tom

Reputation: 185

AWS CLI search a file in s3 bucket and copy to different folder

I am trying to copy only files from AWS S3 Folder_Test1 folder to a Folder_Test2 folder in the same bucket.

Folder_Test1:

T1_abc_june21.csv
T1_abc_june25.csv
T2_abc_june29.csv
T1_abc_def_june21.csv
T2_abc_def_june25.csv
T3_abc_def_june29.csv
T3_xyz_june29.csv

I have to filter the file name having only abc and exclude the files abc_def:

I tried:

aws s3 cp  s3://$bucket/Folder_Test1/ s3://$bucket/Folder_Test2/ --exclude "*abc_def*" --include "*abc*"

but it is not working.

Upvotes: 0

Views: 864

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 270154

From s3 — AWS CLI 1.18.123 Command Reference:

Any number of these parameters can be passed to a command. You can do this by providing an --exclude or --include argument multiple times, e.g. --include ".txt" --include ".png". When there are multiple filters, the rule is the filters that appear later in the command take precedence over filters that appear earlier in the command.

Therefore, the problem is that your command is excluding *abc_def* but is then including *abc*, which adds the *abc_def* files again.

You should be able to fix it by swapping the order:

aws s3 cp  s3://$bucket/Folder_Test1/ s3://$bucket/Folder_Test2/ --include "*abc*" --exclude "*abc_def*"

If it is copying other files that you do not want (eg xyz), then add an exclude:

aws s3 cp  s3://$bucket/Folder_Test1/ s3://$bucket/Folder_Test2/ --exclude "*" --include "*abc*" --exclude "*abc_def*"

This will apply these rules in order:

  • Exclude everything
  • Add *abc*
  • Exclude *abc_def*

Upvotes: 2

Related Questions