CODEWITHSUNDEEP
amazon-s3amazon-iamaws-glue
vr00n
vr00n

Reputation: 596

Getting AWS Glue to write to an S3 bucket in another AWS account

I need to have AWS Glue (Account:PROD) to write to an S3 Bucket on another account (Account:DEV)

According to http://docs.aws.amazon.com/glue/latest/dg/access-control-overview.html

Resource-Based Policies

Other services, such as Amazon S3, also support resource-based permissions policies. For example, you can attach a policy to an S3 bucket to manage access permissions to that bucket. AWS Glue doesn't support resource-based policies.

...which means that I cannot do arn:aws:s3::DEV-Account:S3-Bucket/*

I tried creating a Trusted entity on the DEV Account with PROD and attached a policy set to access the s3 bucket on the DEV account.

How do I go about this?

Upvotes: 7

Views: 11529

Answers (3)

Lydon
Lydon

Reputation: 329

Just an update that Glue now supports Resource Level Policies, but currently only for DataCatalog resources. https://docs.aws.amazon.com/glue/latest/dg/glue-resource-policies.html

Upvotes: 1

vr00n
vr00n

Reputation: 596

We were able to get around this by having the GLUE Job add an ACL to the object it was creating and uploading to the S3 bucket

ACL = {
u'Owner': {u'DisplayName': 'prod', u'ID': 'XXXX'
},
u'Grants': [{
    u 'Grantee': {
        u 'Type': 'CanonicalUser',
        u 'DisplayName': 'prod',
        u 'ID': 'XXXXX'
    },
    u 'Permission': 'FULL_CONTROL'
},
{
    u 'Grantee': {
        u 'Type': 'CanonicalUser',
        u 'DisplayName': 'dev',
        u 'ID': 'YYYY'
    },
    u 'Permission': 'READ'
},
{
    u 'Grantee': {
        u 'Type': 'CanonicalUser',
        u 'DisplayName': 'dev',
        u 'ID': 'YYYY'
    },
    u 'Permission': 'READ_ACP'
},
{
    u 'Grantee': {
        u 'Type': 'CanonicalUser',
        u 'DisplayName': 'dev',
        u 'ID': 'YYYY'
    },
    u 'Permission': 'WRITE_ACP'
}
]
response = client.put_object_acl(Bucket='BUCKET', Key='OBJECT', AccessControlPolicy=ACL)

Upvotes: 0

hoaxz
hoaxz

Reputation: 751

We had the same issue and we came to a solution by adding these into our DEV bucket policies:

{
    "Sid": "SID",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::[PROD-ACCOUNT-ID]:role/[PROD-GLUE-ROLE]"
    },
    "Action": [
        "s3:Get*",
        "s3:Put*",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions",
        "s3:ListMultipartUploadParts"
    ],
    "Resource": [
        "arn:aws:s3:::[DEV-BUCKET]",
        "arn:aws:s3:::[DEV-BUCKET]/*"
    ]
}

And this to the PROD Glue role ([PROD-GLUE-ROLE]) account policies:

{
    "Action": [
        "s3:Get*",
        "s3:List*"
        "s3:Put*"
    ],
    "Resource": [
        "arn:aws:s3:::[DEV-BUCKET]*"
    ],
    "Effect": "Allow"
}

After that you should be able to read and write data from and to your DEV bucket using your PROD role in the PROD account:

data_frame = glue_context.create_dynamic_frame_from_options(
    connection_type='s3',
    connection_options={
        'paths':'s3://[DEV-BUCKET]/...'
    },
    format='json'
)

Hope this helps

Upvotes: 8

Related Questions