Build AWS IAM policy from JSON array of account numbers

Question

I have a list of AWS account numbers that I want to dynamically use to build an AWS policy. Here is my example:

resource "aws_s3_bucket" "splunk-config-bucket" {
  bucket        = "${var.config_bucket_name}"
  force_destroy = true
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = "${data.aws_kms_alias.kms_name.arn}"
        sse_algorithm     = "aws:kms"
      }
    }
  }
  tags = {
    Product = "Splunk - AWS Config Logs"
    Service = "Security"
  }

 lifecycle_rule {
    id      = "log"
    enabled = true

    prefix = "*"

    transition {
      days          = 7
      storage_class = "GLACIER"
    }

    expiration {
      days = 14
    }
  }

  policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSConfigAclCheck20150319",
            "Effect": "Allow",
            "Principal": {
                "Service": "config.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::${var.config_bucket_name}"
        },
        {
            "Sid": "AWSConfigWrite20150319",
            "Effect": "Allow",
            "Principal": {
                "Service": "config.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*”,
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*",
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*",
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*",
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*",
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*",
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*",
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*",
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*",
                "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/123456789/*"
            ],
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}
POLICY
}

resource "aws_s3_bucket_notification" "configlogs_bucketnotification" {
  bucket = "${var.config_bucket_name}"

  queue {
    queue_arn     = "${aws_sqs_queue.splunk_configlogs_sqs_queue.arn}"
    events        = ["s3:ObjectCreated:*"]
  }
}

I have a list of account numbers in an envs.json file, it looks like this:

{
    "accounts": [
        "1234567890",
        "0987654321",
        "1029384756",
        "6574839201",
        "0192837465"
    ]
}

I'm trying to generate the policy with a dynamic list of AWS account numbers. We're trying to get away from hard-coding account numbers and pulling this from a "centralised config" project we have.

Here is what I've tried:

variables.tf:

locals {
    accounts = jsondecode(file("../configuration/envs.json")).accounts
}

Config.tf

            "Principal": {
                "Service": "config.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": ${jsonencode(formatlist("arn:aws:s3:::${var.config_bucket_name}/AWSLogs/%s/*", local.accounts))}
            "Condition": {

This gives me the following output when I run "terraform plan":

Error: "policy" contains an invalid JSON: invalid character '"' after object key:value pair

  on central_config.tf line 2, in resource "aws_s3_bucket" "splunk-config-bucket":
   2: resource "aws_s3_bucket" "splunk-config-bucket" {

I've also tried this:

variables.tf

locals {
  accounts = jsondecode(file("../configuration/envs.json")).accounts
}

output "example" {
  value = jsonencode(formatlist("arn:aws:s3:::${var.config_bucket_name}/AWS/AWSLogs/%s/*", local.accounts))
}

config.tf

            "Principal": {
                "Service": "config.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": ${local.accounts.output.value}
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }

This is the output I get:

Error: Unsupported attribute

  on central_config.tf line 60, in resource "aws_s3_bucket" "splunk-config-bucket":
  60:             "Resource": ${local.accounts.output.value}
    |----------------
    | local.accounts is tuple with 35 elements

This value does not have any attributes.

Thanks,

Answer 1

Getting JSON encoding right with string templates can be frustrating. Unless there's a strong reason to format the JSON a particular way, we can avoid templating JSON by constructing the desired data structure directly and then passing it to jsonencode, like this:

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AWSConfigAclCheck20150319"
        Effect = "Allow"
        Principal = {
          Service = "config.amazonaws.com"
        }
        Action   = "s3:GetBucketAcl"
        Resource = "arn:aws:s3:::${var.config_bucket_name}"
      },
      # etc, etc
    ]
  })

You can then use Terraform's operators and functions directly to construct parts of that object. For example, to produce the list of ARNs you wanted:

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      # ...
      {
        Sid    = "AWSConfigWrite20150319"
        Effect = "Allow"
        Principal = {
          Service = "config.amazonaws.com"
        }
        Action   = "s3:PutObject"
        Resource = [for acct in local.accounts : "arn:aws:s3:::${var.config_bucket_name}/AWSLogs/${acct}/*"]
      },
      # ...
    ]
  })

If the policy becomes complicated enough that you want to factor it out into a separate template file, you can still use the jsonencode function by making the entire template be a jsonencode call:

${jsonencode({
  # ...
})}

Using jsonencode for the whole structure at once means the result is guaranteed to be valid JSON, without needing to worry about exactly where to place commas and other delimiters as you would need to when constructing JSON as a template.

Answer 2

I've managed to get this working with help from ydaetskcoR:

Here is my variables file:

locals {
  accounts = jsondecode(file("../configuration/envs.json")).accounts
  config = jsonencode(formatlist("arn:aws:s3:::${var.config_bucket_name}/AWSLogs/%s/*", local.accounts))
}

I could test the output by using Terraform console:

> jsonencode(formatlist("arn:aws:s3:::${var.config_bucket_name}/AWSLogs/%s/*", local.accounts))
["arn:aws:s3:::bby-central-configlogs-splunk/AWSLogs/123456789/*","arn:aws:s3:::bby-central-configlogs-splunk/AWSLogs/123456789/*"]

As you can see it has [] around them: I put the following into my policy:

            "Principal": {
                "Service": "config.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": ${local.config},
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }

I thought I would put my answer :)

Answer 3

You can load the file using the file function and then extract the list of accounts that you'll need by decoding the JSON with jsondecode and then selecting the accounts key.

As an example:

locals {
  accounts = jsondecode(file("accounts.json")).accounts
}

output example {
  value = local.accounts
}

This will return the following:

example = [
  "1234567890",
  "0987654321",
  "1029384756",
  "6574839201",
  "0192837465",
]

If you then wanted to put that into your policy you'd want to use the formatlist function to pass a list of accounts into a single string format:

output "example" {
  value = formatlist("arn:aws:s3:::bucket_name/AWSLogs/%s/*", local.accounts)
}

This outputs:

example = [
  "arn:aws:s3:::bucket_name/AWSLogs/1234567890/*",
  "arn:aws:s3:::bucket_name/AWSLogs/0987654321/*",
  "arn:aws:s3:::bucket_name/AWSLogs/1029384756/*",
  "arn:aws:s3:::bucket_name/AWSLogs/6574839201/*",
  "arn:aws:s3:::bucket_name/AWSLogs/0192837465/*",
]

If you notice carefully though, Terraform uses trailing commas in lists which is invalid JSON so would create an invalid JSON structure for your IAM policy. To fix that we can then re-encode it into JSON with jsonencode:

output "example" {
  value = jsonencode(formatlist("arn:aws:s3:::bucket_name/AWSLogs/%s/*", local.accounts))
}

Which then outputs:

example = ["arn:aws:s3:::bucket_name/AWSLogs/1234567890/*","arn:aws:s3:::bucket_name/AWSLogs/0987654321/*","arn:aws:s3:::bucket_name/AWSLogs/1029384756/*","arn:aws:s3:::bucket_name/AWSLogs/6574839201/*","arn:aws:s3:::bucket_name/AWSLogs/0192837465/*"]

without the trailing comma.

Putting this altogether you can then create your IAM policy like this:

locals {
  accounts = jsondecode(file("accounts.json")).accounts
}

resource aws_iam_policy policy {
  name        = "example"
  path        = "/"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": ${jsonencode(formatlist("arn:aws:s3:::bucket_name/AWSLogs/%s/*", local.accounts))}
    }
  ]
}
EOF
}

Answer 4

You could have a dynamic config reload in your code that updates the list of accounts and calls S3 API when the config file changes.

For the purpose of watching on config file (account list) you can use some libraries. I believe "Viper" is a good choice.

And also if you are worried about losing these kind of events on file caused by API call failures or application crash, you can try to make updating the action list idempotent and do it on application start or in an interval.

These links might come in hand:

About idempotency: https://nordicapis.com/understanding-idempotency-and-safety-in-api-design/

Viper: https://github.com/spf13/viper