CODEWITHSUNDEEP
amazon-web-servicesterraformterraform-provider-awsaws-lake-formation
DataEnginerd
DataEnginerd

Reputation: 61

LakeFormation permissions when deploying w/ Terraform?

I'm working on deploying to LakeFormation via Terraform. Specifically, granting data location access to a lambda role. I'm getting an error when the role/user I'm deploying with in Terraform isn't an admin on LakeFormation (I haven't tried playing around w/ granular policies on the caller yet). Has anyone come across the same issue and what was the resolution? The caller is a service user which is used by other groups across the org, so I would ideally like to avoid elevating any more of its permissions.

resource "aws_lakeformation_permissions" "datalake-permissions" {
  principal   = aws_iam_role.lambda-role.arn
  permissions = ["DATA_LOCATION_ACCESS"]

  data_location {
    arn = data.aws_s3_bucket.datalake-bucket.arn
  }
}

This is the error :

error creating Lake Formation Permissions (input: { Permissions: ["DATA_LOCATION_ACCESS"], Principal: { DataLakePrincipalIdentifier: "arn:aws:iam::{account_id}:role/lambda_role" }, Resource: { DataLocation: { ResourceArn: "arn:aws:s3:::{my-bucket}" } } }): AccessDeniedException: Resource does not exist or requester is not authorized to access requested permissions.

Also made sure the bucket exists and isn't an issue.

I've tried adding the lambda role as an admin to the space, had no effect. Only appears to be when I give the caller access that things work as expected.

Upvotes: 0

Views: 2076

Answers (2)

Shrey
Shrey

Reputation: 1

Owner role/user of a resource has SUPER permission on the said resource. You can either use a Data Lake Admin Role or Owner Role to grant lake formation permission such as "DATA_LOCATION_ACCESS" to other principals. I haven't tried this with terraform but this works successfully via AWS CLI.

Upvotes: 0

tekneee
tekneee

Reputation: 691

Unfortunately LakeFormation errors are still lacking a lot. I struggled with this issue and I can make two observations:

  1. you need the lakeformation:RegisterResource IAM permissions to register a Data Location. There are a few others that might be required, unfortunately AWS will not report the usual IAM errors you might be used to such as "AccessDenied, you are missing s3:PutObject permissions".
  2. however, unless you are a Data Lake Admin (Lake Formation -> Data Lake Settings), you won't be able to grant this data location to other users/roles/groups/accounts. This is relevant because if you have a Glue Job with Hudi or a Glue Crawler that needs to create tables that point to this Data Location and such Glue Tables will be Lake Formation governed (not having IAMAllowedPrincipals group in its list of permissions), it will have access denied unless it has be granted access to the Data Location.

There are other requirements to register a data location and successfully query it later. For example, if your data in S3 is encrypted, you need to make sure that you have given a KMS grant to the Lake Formation service principal you have assigned to the Data Location (I recommend using a custom one where you programmatically append KMS and S3 permissions as needed).

Upvotes: 1

Related Questions