CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-ec2aws-clirsync
HartleySan
HartleySan

Reputation: 7810

How do I sync an AWS S3 bucket with files on a remote non-AWS server?

I want to sync an AWS S3 bucket up with files on a remote non-AWS server. I have proper access to both the remote server and the EC2 instance that has access to the S3 bucket. What is the best way to do this?

I looked at the docs for the aws s3 sync command, and it looks like you can only sync up an S3 bucket with files locally on the server that has access to the S3 bucket.

The problem is, I have files on a remote server that I want to sync up with an S3 bucket, but that server is not an AWS EC2 instance.

I am able to use the rsync command to get the files from the remote server onto the AWS server that has access to the S3 bucket, but if I do the rsync command and then the aws s3 sync command, then it becomes a two-step process to move the files, takes about twice as long, and because there are a lot of files, I'd also have to increase the size of the EC2 instance volume to do all the files at once. All not ideal.

As such, is there a way to sync up an S3 bucket with a remote server that is not an AWS server and that does not have access to the S3 bucket by using an EC2 instance that does have access to the S3 bucket as an intermediary? Thank you.

Upvotes: 0

Views: 1498

Answers (1)

guest
guest

Reputation: 627

The simplest approach would be to use sshfs. This link has detailed instructions, but the basic process is as follows:

  1. Create a local directory where you'll mount the remote system, such as /tmp/syncmount
  2. Run sshfs USER@REMOTE:DIRECTORY /tmp/syncmount
  3. Run s3sync /tmp/syncmount s3://YOUR_BUCKET

I'm assuming that if you have rsync access you'll have general SSH access.

Upvotes: 2

Related Questions