CODEWITHSUNDEEP
fileamazon-s3amazon-web-servicestransfer
Andrew
Andrew

Reputation: 43153

Move files directly from one S3 account to another?

Pretty basic question but I haven't been able to find an answer. Using Transit I can "move" files from one S3 bucket on one AWS account to another S3 bucket on another AWS account, but what it actually does is download the files from the first then upload them to the second.

Is there a way to move files directly from one S3 account to another without downloading them in between?

Upvotes: 92

Views: 90056

Answers (12)

Chipmonkey
Chipmonkey

Reputation: 865

The given answers so far all require an account that has access to both the source and target s3 buckets. I've found myself recently in a situation where this was not allowed (for various non-technical company reasons that we'll just assume were good).

The solution I ended up going with was to:

  1. Spin up an EC2 instance with permission to write to the target bucket (you can do this locally but the bandwidth and network i/o out of AWS makes EC2 worth it - any tiny instance will do)
  2. Mount the destination folder with s3fs somewhere (/mnt/target)
  3. Give my command line read access (via AWS_ACCESS_KEY_ID, etc) to the source bucket
  4. Use aws sync s3://source_bucket/folder /mnt/target/folder ... (Or mv or cp as needed)

This is the easiest way I've seen to copy between folders when it's not allowed to have a single IAM role with permission to both, and when it's prohibitive to use an intermediate location.

Upvotes: 0

Mohmadhaidar devjiyani
Mohmadhaidar devjiyani

Reputation: 587

Yes, you can transfer the whole s3 bucket from your root account to another AWS root account.

I have tried the given options but they didn't work for me, even I explore solutions from blogs, but that also didn't work for me. So I started exploring properties and the permission tab in the s3 bucket.

And at last, I find one solution which is very easy to achieve and we do not need to create any IAM role or any policy. Just follow the given steps.

Prerequisites:

  • AWS cli Installed and configured
  • S3 bucket created on both source and destination account

Steps:

  • Navigate to the destination s3 bucket and click on permission tab
  • Scroll down to Access Control List (ACL) and click on Edit button
  • Scroll down to Access for other AWS accounts and click on Add grantee button
  • Enter your canonical ID in the textbox and check the object read and write permission box
  • You can get the canonical ID by clicking on the top right corner of the window at your aws account name -> click on security credentials -> On that page, you can copy your canonical id.
  • After adding the grantee click on the Save Changes button
  • Now open your terminal/cmd and fire below command

aws s3 cp --recursive s3://source-bucket s3://destination-bucket --source-region source-region --region destination-region --acl bucket-owner-full-control

This command will do copy and paste operation but if you want to move then you can use mv instead of cp in above command

Here you can replace source-bucket with your actual bucket name from where you want to copy and replace destination-bucket with your actual bucket name where you want to copy.

You can also specify source and destination region name

You can use your machine to do this or you can spin up one ec2 instance and transfer your s3 data.

Upvotes: 1

svikramjeet
svikramjeet

Reputation: 1945

One can so it with running following :

aws s3 mv (sync for keeping buckets in sync) s3://source-bucket s3://destination-bucket --recursive

  1. Attach a bucket policy to the source bucket in Source Account.

  2. Attach an AWS Identity and Access Management (IAM) policy to a user or role in Destination Account.

  3. Use the IAM user or role in Destination Account to perform the cross-account move.

Upvotes: 0

Anand Tripathi
Anand Tripathi

Reputation: 16166

Move S3 files from One account to another account

Let's consider there are two accounts source account and destination account. And two buckets source-bucket and destination bucket. We want to move all files from source-bucket to destination-bucket. We can do it by the following steps:

  1. aws configure
    • Configure your destination account using the credential or the IAM role.
  2. Create user policy for the destination account user.
  3. Give destination user access to the source-bucket by modifying the source-bucket policy and adding destination account user policy into it. By this way, destination user will have the access to source-bucket.
  4. aws s3 ls s3://source-bucket/
    • this will check whether the destination account is having access to source-bucket. Just for confirmation do this.
  5. aws s3 cp s3://source-bucket s3://destination-bucket --recursive
    • this will copy source-bucket all files to destination-bucket. All files are copied using --recursive flag.
  6. aws s3 mv s3://source-bucket s3://destination-bucket --recursive
    • this will move all the files from source-bucket to destination-bucket.

Alternative you can use the sync command - aws s3 sync s3://source-bucket s3://detination-bucket

For Better Explanation follow the link

Upvotes: 4

Marco Delavegua
Marco Delavegua

Reputation: 61

You can user Cyberduck (open source)

Upvotes: 2

curious_george
curious_george

Reputation: 642

For newly created files (NOT existing objects), you can take advantage of new functionality from AWS. It is Cross-Region Replication (under "Versioning" for the S3 bucket). You can create a policy that will allow you to replicate new objects to a bucket in a different account.

For existing objects, you will still need to copy your objects using another method - unless AWS introduces native functionality for this in the future.

Upvotes: 1

Programster
Programster

Reputation: 12814

Use the aws cli (I used ubuntu 14 ec2 instance) and just run the following command:

aws s3 sync s3://bucket1 s3://bucket2

You will need to specify the account details for one, and have public write access or public read access to the other.

This will sync the two buckets. You can use the same command again later to sync quickly. Best part is that it doesn't seem t require any bandwidth (e.g. files are not passing through local computer).

Upvotes: 60

One Giant Leap AB
One Giant Leap AB

Reputation: 341

On Mac OS X I used the Transmit app from Panic. I opened one window for each S3 account (using the API Keys and secrets). I could then drag from one bucket in one window to another bucket in the other window. No need to download files locally first.

Andrew is correct, Transmit downloads the files locally then uploads the files.

Upvotes: 3

Gatorhall
Gatorhall

Reputation: 411

CrossFTP can copy S3 files straight from one bucket to another without downloading them. It is a GUI S3 client that works on Windows, Mac, and Linux.

Upvotes: 2

side2k
side2k

Reputation: 2074

Yes, there is a way. And its pretty simple, though it's hard to find it. 8)

For example, suppose your first account username is [email protected] and second is [email protected].

Open AWS Management Console as acc1. Get to the Amazon S3 bucket properties, and in the "Permissions" tab click "Add more permissions". Then add List and View Permissions for "Authenticated Users".

Next, in AWS IAM (it's accessible from among the console tabs) of acc2 create a user with full access to the S3 bucket (to be more secure, you can set up exact permissions, but I prefer to create a temporary user for the transfer and then delete it).

Then you can use s3cmd (using the credentials of the newly created user in acc2) to do something like:

s3cmd cp s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive

All transfer will be done on Amazon's side.

Upvotes: 125

Rose Perrone
Rose Perrone

Reputation: 63586

boto works well. See this thread. Using boto, you copy objects straight from one bucket to another, rather than downloading them to the local machine and uploading them to another bucket.

Upvotes: 6

Geoff Appleford
Geoff Appleford

Reputation: 18832

If you are just looking for a ready made solution there are a few solutions out there that can do this. Bucket Explorer works on Mac and Windows and can copy across accounts as can Cloudberry S3 Explorer and S3 Browser but they are Windows only so may not work for you.

I suspect the AWS console could also do it with the appropriate permissions setup but I haven't tested this.

You can also do it using the AWS API as long as you have given the AWS account you are using write permissions to the destination bucket.

Upvotes: 22

Related Questions