CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-ec2aws-java-sdkamazon-kms
LMK IND
LMK IND

Reputation: 502

How to access AWS S3 with KMS from org.apache.hadoop.fs.s3a.S3AFileSystem API

I have EC2 configured with

  <property>
     <name>fs.s3a.aws.credentials.provider</name>
    <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value>
  </property>
 <property>
    <name>fs.s3a.server-side-encryption-algorithm</name>
    <value>SSE-KMS</value>
  </property>
 <property>
    <name>fs.s3a.server-side-encryption.key</name>
    <value>arn:aws:kms:zz-jjbbcc-1:123432:key/AABBCC</value>
  </property>

with this configuration in core-site.xml I am easily able to put files and create folders (via aws cli) in S3 without giving any authentication details as I have already configured in core-site.xml

Now I want to access the S3 bucket and create folder by using S3AFileSystem, But when i am calling

S3AFileSystem fs  
fs.mkdirs(somepath); // this will create folder in S3. 
Its throwing Accessdenied 403 Exception. 


java.nio.file.AccessDeniedException: s3a://xxx-xxx/xxx/.FolderIwantToCreate: innerMkdirs on s3a://xxx-xxx/xxx/.FolderIwantToCreate: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XXXXXXX; S3 Extended Request ID: xxxxxxxxxxxxx=), S3 Extended Request ID: /XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:174) ~[hadoop-aws-2.9.2.jar:?]
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:117) ~[hadoop-aws-2.9.2.jar:?]
    at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1683) ~[hadoop-aws-2.9.2.jar:?]
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2216) ~[hadoop-common-2.9.2.jar:?]

But while debugging, I tried

((S3AFileSystem) fs).delete(path,true)

where path has s3://MyBUCKET/SOMEFOLDER/ surprisingly SOMEFOLDER gets deleted

Did i miss any configuration? S3(enabled with KMS) and doesn't have any bucket policy, IAM has Full Access on S3.

EC2 has IAM Role which has the following policies:
S3 Access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

KMS Policy :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:CreateAlias",
                "kms:CreateKey",
                "kms:DeleteAlias",
                "kms:Describe*",
                "kms:GenerateRandom",
                "kms:Get*",
                "kms:List*",
                "kms:TagResource",
                "kms:UntagResource",
                "iam:ListGroups",
                "iam:ListRoles",
                "iam:ListUsers"
            ],
            "Resource": "*"
        }
    ]
}

S3 Bucket Don't have any Policies.
S3 Permission

Upvotes: 3

Views: 710

Answers (1)

Foghorn
Foghorn

Reputation: 2326

I would look at your KMS Key policy. There are two things I see:

  1. Key policies are 'king'. They override even IAM policies when it comes to that particular key. This key policy lacks the ability to use IAM to delegate/give KMS permissions on this key. With this policy, even if an IAM policy would give permission to use this particular KMS key, they would be ignored although there would be no indication of that from the IAM service. See the example under 'Default Key Policy' here for more information: https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html

  2. If you are storing keys in S3, I think you need kms:GenerateDataKey, and possibly kms:decrypt permission depending on how the service will verify it successfully wrote the file. Sometimes, upon setting this up the underlying service will quickly write and read back a temporary file you never see to make sure the permissions are in place. See somewhat related link here for what S3 permissions are needed from KMS when using that service for CMKs: https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-access-default-encryption/

Upvotes: 1

Related Questions