CODEWITHSUNDEEP
bashshellamazon-web-servicesaws-clijenkins-cli
MichealMills
MichealMills

Reputation: 335

how to exclude the folder from aws s3 sync command

We have an aws s3 sync command that sync's the data from bitbucket to s3 via Jenkins execute shell. Below is the s3 bucket structure

cdn-accountname > qa > sitename > cdn > img
cdn-accountname > qa > sitename > cdn > css
cdn-accountname > qa > sitename > cdn > png

We use below command to sync the cdn content from bitbucket to s3

aws s3 sync cdn/ s3://cdn-accountname/qa/sitename

I would like to exclude "/upload" and "/upload_qa" folders within cdn/ folder and sync it to s3. I tried below commands but none of them worked

aws s3 sync --exclude "cdn/upload" --exclude "cdn/upload_qa" cdn/ s3://cdn-accountname/qa/sitename
aws s3 sync --exclude=cdn/upload --exclude=cdn/upload_qa cdn/ s3://cdn-accountname/qa/sitename
aws s3 sync --exclude "*/upload/*" --exclude "*/upload_qa/*" cdn/ s3://cdn-accountname/qa/sitename
aws s3 sync --exclude 'upload/*' --exclude 'upload_qa/*' cdn/ s3://cdn-accountname/qa/sitename 
aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude 'upload/*' --exclude 'upload_qa/*'
aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude "cdn/upload" --exclude "cdn/upload_qa"
aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude=cdn/upload --exclude=cdn/upload_qa
aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude "*/upload/*" --exclude "*/upload_qa/*"

suggest the workable s3 sync command to sync the folder excluding sub-folder

Upvotes: 6

Views: 9126

Answers (1)

maafk
maafk

Reputation: 6876

I believe you'll need to also use the --include flag in your command (see docs for reference)

aws s3 sync --include "*" --exclude "upload/*" --exclude "upload_qa/*" cdn/ s3://cdn-accountname/qa/sitename

Be careful the order

When there are multiple filters, the rule is the filters that appear later in the command take precedence over filters that appear earlier in the command

Tested this out and working fine

home [~]: aws s3 ls s3://abc-fe-testing1
                           PRE folder1/
                           PRE folder2/
                           PRE folder3/
                           PRE folder4/
home [~]: aws s3 ls s3://abc-fe-testing2
home [~]: aws s3 sync s3://abc-fe-testing1 s3://abc-fe-testing2 --exclude "folder1*" --exclude "folder2*"
copy: s3://abc-fe-testing1/folder4/file_from_folder4.txt to s3://abc-fe-testing2/folder4/file_from_folder4.txt
copy: s3://abc-fe-testing1/folder3/file_from_folder3.txt to s3://abc-fe-testing2/folder3/file_from_folder3.txt
home [~]: aws s3 ls s3://abc-fe-testing2
                           PRE folder3/
                           PRE folder4/

Upvotes: 3

Related Questions