CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
chasez0r
chasez0r

Reputation: 634

AWS CLI S3 Bucket to Bucket copying?

While performing an aws s3 cp --recursive s3://src-bucket s3://dest-bucket command, will it download the files locally and upload them to the destination bucket? Or (hopefully) will this entire transaction happen on AWS without files ever hitting your instantiating machine?

Thanks

Upvotes: 3

Views: 696

Answers (1)

jarmod
jarmod

Reputation: 78860

The copy happens within AWS. I verified this as follows using awscli on an Ubuntu EC2 instance:

  1. upload 4GB of files to bucket1: peak 140 mbps sent, real time 45s, user time 32s
  2. sync bucket1 to bucket2: peak 60 kbps sent, real time 22s, user time 2s

Note: 'real' time is wall clock time, 'user' time is CPU time in user mode.

So, there is a significant difference in peak bandwidth used (140mbps vs 60kbps) and in CPU usage (32s vs. 2s). In case #1 we are actually uploading 4 GB of files to S3 but in case #2 we are copying 4 GB of files from one S3 bucket to another without them touching our local machine. The small amount of bandwidth used in case #2 is related to the awscli displaying progress of the sync.

I saw basically identical results when coping objects (aws s3 cp) as when syncing objects (aws s3 sync) between S3 buckets.

Upvotes: 6

Related Questions