CODEWITHSUNDEEP
amazon-s3aws-cli
ZN13
ZN13

Reputation: 1108

Downloading from S3 with aws-cli using filter on specific prefix

For some reason there's a bucket with a bunch of different files, all of which have the same prefix but with different dates:

backup.2017-01-01aa

backup.2017-01-01ab

backup.2017-01-15aa

backup.2017-01-15ab

backup.2017-02-01aa

backup.2017-02-01ab

etc..

How do I download only files that start with "backup.2017-01-01"?

Upvotes: 24

Views: 20949

Answers (3)

jvian
jvian

Reputation: 83

A PowerShell equivalent for @sampo-smolander's answer

,@(Get-ChildItem -recurse | aws s3 ls s3://yourbucket/backup.2017-) | Select-Object -ExpandProperty syncroot |  foreach-Object {$_.split(" ")[-1]} | %{&"aws" s3 cp s3://yourbucket/$_ .}

Posting it here since I spent a lot of time figuring out this, so hopefully it helps someone else who needs to use powershell. Also I'm not too familiar with powershell so it may need optimization.

Upvotes: 1

Sampo Smolander
Sampo Smolander

Reputation: 1756

I think --include does the filtering locally. So if your bucket contains millions of files, the command can take hours to run, because it needs to download a list of all the filenames in the bucket. Also, some extra network traffic.

But aws s3 ls can take a truncated filename to list all the corresponding files, without any extra traffic. So you can

aws s3 ls s3://yourbucket/backup.2017-

to see your files, and something like

aws s3 ls s3://yourbucket/backup.2017- | colrm 1 31 | xargs -I % aws s3 cp s3://yourbucket/% .

to copy your files.

Upvotes: 43

ZN13
ZN13

Reputation: 1108

You'll have to use aws s3 sync s3://yourbucket/

There are two parameters you can give to aws s3 sync; --exclude and --include, both of which can take the "*" wildcard.

First we'll have to --exclude "*" to exclude all of the files, and then we'll --include "backup.2017-01-01*" to include all the files we want with the specific prefix. Obviously you can change the include around so you could also do something like --include "*-01-01*".

That's it, here's the full command:

aws s3 sync s3://yourbucket/ . --exclude "*" --include "backup.2017-01-01*"

Also, remember to use --dryrun to test your command and avoid downloading all files in the bucket.

Upvotes: 29

Related Questions