CODEWITHSUNDEEP
amazon-web-servicesamazon-s3s3distcp
Kishor Baindoor
Kishor Baindoor

Reputation: 111

Copying S3 Files across AWS account using s3-dist-cp

I have requirement where I need to copy files from one S3 bucket to other S3 bucket. These buckets are present in different AWS account. I tried using s3 sync command. But, for this destination IAM user should be given with read access on source bucket. I am not using this command because I don't have same same IAM user with access to both the accounts. There are two AIM user as of now, one for each.

Is there anyway I can use s3-dist-cp command for this requirement?

Upvotes: 2

Views: 2077

Answers (1)

Kannaiyan
Kannaiyan

Reputation: 13055

As of this writing, you cannot configure two different accesskeys for two different buckets.

You can configure the same IAM user across accounts and allow them to read from one and write to another. IAM roles can make it happen here.

If you don't want to use the same IAM user across accounts copy to local hdfs and move back to destination bucket.

hadoop distcp -Dfs.s3a.awsAccessKeyId= -Dfs.s3a.awsSecretAccessKey= s3a://sourcebucket hdfs://tempcopy/

hadoop distcp -Dfs.s3a.awsAccessKeyId= -Dfs.s3a.awsSecretAccessKey= hdfs://tempcopy/ s3a://destbucket

Hope it helps.

Upvotes: 4

Related Questions