Stockburn
Stockburn

Reputation: 115

Copy Data From AWS S3 Bucket Locally Based on Date of File

I want to copy the Latest CSV file which has the date appended from an AWS S3 bucket to a local drive.

I have the basic code that will download the file but it downloads all the files in the bucket I only want the file uploaded that day, latest file.

Upvotes: 1

Views: 4166

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 269101

Download latest object by modified date

If you only wish to grab the file that was last stored on Amazon S3, you could use:

aws s3 cp s3://my-bucket/`aws s3api list-objects-v2 --bucket my-bucket --query 'sort_by(Contents, &LastModified)[-1].Key' --output text` .

This command does the following:

  • The inner aws s3api list-objects-v2 command lists the bucket, sorts by date (reversed), then returns the Key (filename) of the object that was last modified
  • The outer aws s3 cp command downloads that object to the local directory

Download latest object based on filename

If your filenames are like:

some_file_20190130.csv
some_file_20190131.csv
some_file_20190201.csv

then you can list by prefix and copy the last one:

aws s3 cp s3://my-bucket/`aws s3api list-objects-v2 --bucket my-bucket --prefix some_file_ --query 'sort_by(Contents, &Key)[-1].Key' --output text` .

This command does the following:

  • The inner aws s3api list-objects-v2 command lists the bucket, only shows files with a given prefix of some_file_, sorts by Key (reversed), then returns the Key (filename) of the object that is at the end of the sort
  • The outer aws s3 cp command downloads that object to the local directory

Upvotes: 3

Related Questions