CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-ec2
Preston Martin
Preston Martin

Reputation: 2963

AWS Synchronize S3 bucket with EC2 instance

I would like to synchronize an S3 bucket with a single directory on multiple Windows EC2 instances. When a file is uploaded or deleted from the bucket, I would like it to be immediately pushed or removed respectively from all of the instances. New instances will be added/removed frequently (multiple times per week). Files will be uploaded/deleted frequently as well. The files sizes could be up to 2gb in size. What AWS services or features can solve this?

Upvotes: 3

Views: 1677

Answers (3)

Deepak Chaudhary
Deepak Chaudhary

Reputation: 486

Edit : This following answer is to sync EC2 with S3 Bucket, Source : EC2 & Destination : Bucket.

If it were for only one instance, then only aws cli sync(with --delete option) would have been worked for both: putting files to S3 bucket and to delete.

But the case here is for Multiple Instances, so if we use aws s3 sync with --delete option, there would be a problem.

To explain it simply, consider Instance I1 with files a.jpg & b.jpg to be synced to Bucket. Now a CRON job has synced the files with the S3 bucket.

Now we have Instance I2 which has files c.jpg & d.jpg. So when the CRON job of this Instance runs, it puts the files c.jpg & d.jpg and also deletes the files a.jpg & b.jpg, because those files doesn't exist in Instance I2.

So to rectify the problem we have two approaches :

  1. Sync all files across all Instances(Costly and removes the purpose of S3 altogether).
  2. Sync files without the --delete option, and implement the deletion separately(using aws s3 rm).

Upvotes: -1

adamkonrad
adamkonrad

Reputation: 7122

Based on what you've described, I'd propose the following solution to this problem.

You need to create an SNS topic for S3 change notifications. Then you need a script that's going to subscribe to this topic from your machines. This script will update files on your machines based on changes coming from S3. It should support basic CRUD operations.

Run this script and then sync contents of your S3 to your machine when it starts using aws-cli mentioned above.

Upvotes: 2

TJCloudmin
TJCloudmin

Reputation: 165

Yes, i have used the aws cli s3 "sync" command to keep a local server's content updated with S3 changes. It allows a local target directory's files to be synchronized with a bucket or prefix. http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

Upvotes: 0

Related Questions