CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-cliboto
user13542716
user13542716

Reputation:

Copy file from the s3 to the local?

I have a lot of files in my s3 bucket, so is there any aws cli command which I can use to find a most recent file with a prefix name in s3? and how can I copy that file from s3 to my local folder? Can I use Boto3 or python library to do this?

Upvotes: 4

Views: 14162

Answers (3)

John Rotenstein
John Rotenstein

Reputation: 270164

Here's show to do it in Python:

import boto3

s3_client = boto3.client('s3')

response = s3_client.list_objects_v2(Bucket='MY-BUCKET', Prefix='foo/')
objects = sorted(response['Contents'], key=lambda obj: obj['LastModified'])

## Latest object
latest_object = objects[-1]['Key']
filename = latest_object[latest_object.rfind('/')+1:] # Remove path

# Download it to current directory
s3_client.download_file('MY-BUCKET', latest_object, filename)

Basically, you get back ALL objects, then sort them by LastModified.

Please note that the list_objects_v2() command only returns a maximum of 1000 objects. If the bucket has more, you'll need to loop or use a paginator. See: Paginators — Boto3 documentation

Upvotes: 8

John Rotenstein
John Rotenstein

Reputation: 270164

This command will list the 'latest' object for a given prefix:

aws s3api list-objects --bucket MY-BUCKET --prefix foo/ --query 'sort_by(Contents, &LastModified)[-1].Key' --output text

You could combine it with a copy command:

key=$(aws s3api list-objects --bucket MY-BUCKET --prefix foo/ --query 'sort_by(Contents, &LastModified)[-1].Key' --output text)
aws s3 cp s3://MY-BUCKET/$key .

The --query parameter is very powerful. See: JMESPath Tutorial

Upvotes: 4

Ashish Bhatia
Ashish Bhatia

Reputation: 629

You can refer to this answer for most recent file. get last modified object from S3 CLI. For prefix to be in the object list you can just use

aws s3 ls $BUCKET --recursive | sort | grep <prefix>

Thanks

Ashish

Upvotes: 0

Related Questions