CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-cli
SeattleDBA
SeattleDBA

Reputation: 21

AWS S3 download and copy

We have a bucket in AWS S3 where backups from production are being copy to.

My task is to copy the most recent backup file from AWS S3 to the local sandbox SQL Server, then do the restore.

I have installed all of the AWS tools for windows on the local server. Credentials to connect to AWS S3 work, etc. My local server can list all of the files in the AWS S3 bucket. I can successfully download a single file if I specifically name that file.

Here is an example of that working pulling the most recent copy from July 25, 2016.

aws s3 cp s3://mybucket/databasefile_20160725.zip E:\DBA

My goal is to have a copy script that only pulls the most recent file, which I won't know the name of. I want to schedule this.

Nothing I google or try is getting me the correct syntax to do this.

Upvotes: 2

Views: 1272

Answers (1)

Frederic Henri
Frederic Henri

Reputation: 53803

to retrieve the latest file in your bucket you can do the following

aws s3api list-objects --bucket "mybucket" |\
jq '.Contents | sort_by(.LastModified) | .[-1].Key' --raw-output

The first command will list the objects of your bucket in Json, the elements of the JSon are listed here Then you want to sort the element from their last modified date, take the last element, and you want they Key (i.e. name of the file in bucket). Adding the --raw-output flag to stripe quotes from the key name

You can reuse that in script or pipe it with the s3 cp command like below

aws s3api list-objects --bucket "mybucket" |\
jq '.Contents | sort_by(.LastModified) | .[-1].Key' --raw-output |\
xargs -I {} aws s3 cp s3://mybucket/{} E:\DBA

Upvotes: 4

Related Questions