CODEWITHSUNDEEP
amazon-web-servicesamazon-s3migrationfile-transferdata-transfer
Thejus A P
Thejus A P

Reputation: 71

Move large volumes (> 50tb) of data from one s3 to another s3 in another account cost effectively

I have some S3 buckets in one AWS account which have large amount of data (50+ Tbs)

I want it to move it to new S3 buckets in another account completely and use the 1st AWS account for another purpose.

The method I know is AWS CLI using s3 cp/s3 sync/s3 mv , but this would take days when running in my laptop And I want it to be more cost effective when considering the data transfer also.

Buckets contain mainly zip files and rar files having size ranging from 1GB to 150+GB and also other files too.

Can someone suggest me methods to do this which would be cost effective as well as less time consuming .

Upvotes: 1

Views: 1418

Answers (2)

swooders
swooders

Reputation: 189

You can use Skyplane which is much faster and cheaper than aws s3 cp (up to 110x for large files). Skyplane will automatically compress data to reduce egress costs, and will also give you cost estimates before running the transfer.

You can transfer data between buckets in region A and region B with:

skyplane cp -r s3://<region-A-bucket>/ s3://<region-B-bucket>/

Upvotes: 2

Haitham Gad
Haitham Gad

Reputation: 1579

If the destination bucket is in the same region as the source bucket (even if it's in a different account), there's no data transfer cost for running s3 cp/sync/mv according to the docs (check the Data transfer tab).

For a fast solution, consider using S3 Transfer Acceleration, but note that this does incur transfer costs.

Upvotes: 1

Related Questions