Terraform: iterate over set of objects and value from object

Question

I have this variable:

variable "offset_counter" {
  description = "Represents topic and consumer group properties"
  type = set(object({
    topic_name              = string
    topic_partitions_number = number
    consumer_group_name     = string
    max_offset_lag          = number
  }))
}

and I want to create a lot of resources based on topic_name and topic_partitions_number. For example, I'll have 2 topics (topic A and topic B), and I'll have 20 partitions for topic A and 50 partitions for topic B. As a result, I want to receive 70 (20+50) resources.

Lets imagine, that we have variable:

offset_counter = [
  {
    topic_name              = "topic_A"
    topic_partitions_number = 2
    consumer_group_name     = "my_consumer_group_A"
    max_offset_lag          = 200
  },
  {
    topic_name              = "topic_B"
    topic_partitions_number = 3
    consumer_group_name     = "my_consumer_group_B"
    max_offset_lag          = 50
  }
]

and as a result I want to receive these resources:

resource "aws_cloudwatch_metric_alarm" "offset-lag-too-high" {
  alarm_name = "Alarm for topic topic_A on partition 1: offset lag is more then 200. Consumer group: my_consumer_group_A"
  ........
}

resource "aws_cloudwatch_metric_alarm" "offset-lag-too-high" {
  alarm_name = "Alarm for topic topic_A on partition 2: offset lag is more then 200. Consumer group: my_consumer_group_A"
  ........
}


resource "aws_cloudwatch_metric_alarm" "offset-lag-too-high" {
  alarm_name = "Alarm for topic topic_B on partition 1: offset lag is more then 50. Consumer group: my_consumer_group_B"
  ........
}

And so on. Is it possible somehow?

Answer 1

Yes, I think it is possible. First we will have to declare a local variable and transform the offset_counter into something on which we can iterate through:

locals {
  topics = { for item in flatten(
    [
      for topic in var.offset_counter : [
        for partition in range(1, topic.topic_partitions_number + 1) :
        {
          "key" : "${topic.topic_name}_${partition}",
          "value" : merge(topic, { "current_partition" : partition })
        }
      ]
  ]) : item["key"] => item["value"] }
}

If we provide the input from your question, the content for this local variable will look something like this:

{
  "topic_A_1" = {
    "consumer_group_name" = "my_consumer_group_A"
    "current_partition" = 1
    "max_offset_lag" = 200
    "topic_name" = "topic_A"
    "topic_partitions_number" = 2
  }
  "topic_A_2" = {
    "consumer_group_name" = "my_consumer_group_A"
    "current_partition" = 2
    "max_offset_lag" = 200
    "topic_name" = "topic_A"
    "topic_partitions_number" = 2
  }
  "topic_B_1" = {
    "consumer_group_name" = "my_consumer_group_B"
    "current_partition" = 1
    "max_offset_lag" = 50
    "topic_name" = "topic_B"
    "topic_partitions_number" = 3
  }
  "topic_B_2" = {
    "consumer_group_name" = "my_consumer_group_B"
    "current_partition" = 2
    "max_offset_lag" = 50
    "topic_name" = "topic_B"
    "topic_partitions_number" = 3
  }
  "topic_B_3" = {
    "consumer_group_name" = "my_consumer_group_B"
    "current_partition" = 3
    "max_offset_lag" = 50
    "topic_name" = "topic_B"
    "topic_partitions_number" = 3
  }
}

You can see that we have a flattened map with key/value pairs. What we have to do is to provide this to the for_each attribute:

resource "aws_cloudwatch_metric_alarm" "foobar" {
  for_each   = local.topics
  alarm_name = "Alarm for topic ${each.value.topic_name} on partition ${each.value.current_partition}: offset lag is more then ${each.value.max_offset_lag}. Consumer group: ${each.value.consumer_group_name}"
...
}