CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-cli
Pep
Pep

Reputation: 2027

aws s3 sync not working as expected

I'm trying to set up a one-way directory sync process from one local PC to an AWS EC2 instance via S3.

Both machines are Windows.

I tried using the command line interface.

On the local machine:

aws s3 sync source_dir s3://bucket --region eu-central-1

This command seems to to work well. If there is nothing new, nothing is sync'ed. So far so good.

On the AWS instance:

aws s3 sync s3://bucket target_dir --region eu-central-1

With this command, I have a an issue. Whenever I run it, there is always something to download (it seems to be always the same set of files, perhaps they are all of them, but it seems a subset of them). My expectation was that once in sync, running the command again produced no downloads.

I granted these permissions in the policy: 

"Action": [
     "s3:GetObject",
     "s3:GetObjectAcl",
     "s3:ListBucket",
     "s3:PutObject",
     "s3:PutObjectAcl" 
],
"Resource": [
     "arn:aws:s3:::bucket_name",
     "arn:aws:s3:::bucket_name/*" 
]

Am I missing anything in this setup so that I do not get files downloaded if there is nothing to download when I run the second sync?

Upvotes: 2

Views: 7056

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 270224

You appear to be doing two separate syncs:

  • From a local machine to Amazon S3
  • From Amazon S3 to an Amazon EC2 instance

The problem might be related to timestamps. Amazon EC2 instances always operate as UTC. This might be different to the originating local machine.

If you run the S3->EC2 sync and then run it again immediately, there should be no files copied the second time. If files ARE copied, try updating your AWS CLI to the latest version. If problems persist, try syncing from EC2->S3 and then try S3->EC2 again.

Upvotes: 1

Related Questions