Reputation: 115
I want to copy the Latest CSV file which has the date appended from an AWS S3 bucket to a local drive.
I have the basic code that will download the file but it downloads all the files in the bucket I only want the file uploaded that day, latest file.
Upvotes: 1
Views: 4166
Reputation: 269101
Download latest object by modified date
If you only wish to grab the file that was last stored on Amazon S3, you could use:
aws s3 cp s3://my-bucket/`aws s3api list-objects-v2 --bucket my-bucket --query 'sort_by(Contents, &LastModified)[-1].Key' --output text` .
This command does the following:
aws s3api list-objects-v2
command lists the bucket, sorts by date (reversed), then returns the Key (filename) of the object that was last modifiedaws s3 cp
command downloads that object to the local directoryDownload latest object based on filename
If your filenames are like:
some_file_20190130.csv
some_file_20190131.csv
some_file_20190201.csv
then you can list by prefix and copy the last one:
aws s3 cp s3://my-bucket/`aws s3api list-objects-v2 --bucket my-bucket --prefix some_file_ --query 'sort_by(Contents, &Key)[-1].Key' --output text` .
This command does the following:
aws s3api list-objects-v2
command lists the bucket, only shows files with a given prefix of some_file_
, sorts by Key (reversed), then returns the Key (filename) of the object that is at the end of the sortaws s3 cp
command downloads that object to the local directoryUpvotes: 3