How to sync multiple S3 buckets using multiple AWS accounts?

Question

I am having trouble syncing two S3 buckets that are attached to two separate AWS accounts.

There are two AWS accounts - Account A which is managed by a third party and Account B, which I manage. I am looking to pull files from an S3 bucket in Account A to an S3 bucket in Account B.

Account A provided me the following instructions:

• In Account B, create a new IAM user called LogsUser. Attach the following policy to the user:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "sts:AssumeRole",
                "Resource": "arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role"
            }
        ]
    }

• Configure the AWS CLI to update the config and credentials files. Specifically, the ~/.aws/config file to look like:

  [profile LogsUser]
  role_arn = arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role
  source_profile = LogsUser
  

  And the ~/.aws/credentials file to look like

  aws_access_key_id = YOUR_ACCESS_KEY_ID
  aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
  

• From here, I am successfully able to query the log files in Account A's bucket using $ aws s3 ls --profile LogsUser s3://bucket-a.

I have set up bucket-b in Account B, however, I am unable to query any files in bucket-b. For example, $ aws s3 ls --profile LogsUser s3://bucket-b returns An error occurred (AccessDenied) when calling the AssumeRole operation: Access denied.

Is there something additional I can add to the config file or my IAM policy to allow access to bucket-b using --profile LogsUser option? I can access bucket-b using other --profile settings, but am not looking to sync to the local file system and then to another bucket.

The desired results is to run a command like aws s3 sync s3://bucket-a s3://bucket-b --profile UserLogs.

Answer 1

For example, if you want to copy “Account A” S3 bucket objects to “Account B” S3 bucket, follow below.

Create a policy for the S3 bucket in “account A” like the below policy. For that, you need “Account B” number, to find the B account number go to Support → Support center and copy the account number from there.

Setup “account A” bucket policy :

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT_B_NUMBER:root"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::ACCOUNT_A_BUCKET_NAME/*",
                "arn:aws:s3:::ACCOUNT_A_BUCKET_NAME"
            ]
        }
    ]
}

Log into “Account B” and create a new IAM user or attach the below policy for the existing user.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::ACCOUNT_A_BUCKET_NAME",
                "arn:aws:s3:::ACCOUNT_A_BUCKET_NAME/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::ACCOUNT_B_BUCKET_NAME",
                "arn:aws:s3:::ACCOUNT_B_BUCKET_NAME/*"
            ]
        }
    ]
}

Configure AWS CLI with “Account B” IAM user(Which you have created IAM with the above user policy)

aws s3 sync s3://ACCOUNT_A_BUCKET_NAME s3://ACCOUNT_B_BUCKET_NAME --source-region ACCOUNT_A_REGION-NAME --region ACCOUNT_B_REGION-NAME

This way we can copy S3 bucket objects over different AWS accounts.

If you have multiple awscli profiles, use --profile end of the command with profile name.

Answer 2

Your situation is:

• You wish to copy from Bucket-A in Account-A
• The files need to be copied to Bucket-B in Account-B
• Account-A has provided you with the ability to assume LogAccess-role in Account-A, which has access to Bucket-A

When copying files between buckets using the CopyObject() command (which is used by the AWS CLI sync command), it requires:

• Read Access on the source bucket (Bucket-A)
• Write Access on the destination bucket (Bucket-B)

When you assume LogAccess-role, you receive credentials that have Read Access on Bucket-A. That is great! However, those credentials do not have permission to write to Bucket-B because it is in a separate account.

To overcome this, you should create a Bucket Policy on Bucket-A that grants Write Access to LogAccess-role from Account-B. The Bucket Policy would look something like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT-A:role/12345-LogAccess-role"
            },
            "Action": [
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-a",
                "arn:aws:s3:::bucket-a/*"
            ]
        }
    ]
}

(You might need other permissions. Check any error messages for hints.)

That way, LogAccess-role will be able to read from Bucket-A and write to Bucket-B.

Answer 3

If you just want to list objects in bucket-b, do this.

First make sure the LogsUser IAM user has got proper permission to access the bucket-b s3 bucket in Account B. You can add this policy to the user if not

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-b/*"
            ]
        }
    ]
}

If there is permissions attached to the user, and if the Access keys and Secret Key stored in ~/.aws/credentials stored as [default] belongs to LogsUser IAM user, you can simply list objects inside bucket-b with following command. aws s3 ls

If you want to run the command aws s3 sync s3://bucket-a s3://bucket-b --profile UserLogs, do this.

Remember, we will be using temporary credentials created by STS after assuming the role with permanent credentials of LogsUser. That means the role in Account A should have proper access to both buckets to perform the action and the bucket(bucket-b) in another account (Account B) should have proper bucket policy to allow the role to perform S3 operations.

To provide permissions to the role to access bucket-b, attach following bucket policy to bucket-b.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNTID:role/12345-LogAccess-role"
            },
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-b/*"
            ]
        }
    ]
}

Also in Account A, attach a policy to the role like below to allow access to S3 buckets in both the accounts.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-b/*",
                "arn:aws:s3:::bucket-a/*"
            ]
        }
    ]
}

Answer 4

I would suggest you to consider you to use AWS S3 bucket replication:

https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html