CODEWITHSUNDEEP
amazon-web-servicesamazon-s3terraformaws-glue
krishna hegde
krishna hegde

Reputation: 51

Terraform: Create a AWS glue table using for each option

I needed to create a multiple glue database of "name" and for each database need to create multiple glue tables present in "tablenames" list.

variable "my_map" {
  type = map(object({
    name       = string
    tablenames = []
  }))

  default = {
    obj1 = {
      name       = "NameA1"
      tablenames = ["TableA1", "TableA2", "TableA3"]
    },
    obj2 = {
      name       = "NameB1"
      tablenames = ["TableB1"]
    },
  }
}

resource "aws_s3_bucket" "my_bucket" {
  for_each = var.my_map
  bucket   = "${lower(each.value.name)}"
}

resource "aws_glue_catalog_database" "my_glue_database" {
  for_each = var.my_map
  name     = "${each.value.name}"
}

Now I need to create the multiple glue table, but how do use for_each to have "database_name " and "location" pointing the resource created previously? Also have suffix to be populated accordingly. location = "s3://${aws_s3_bucket.my_bucket[each.key].id}/suffix/" ##### here suffix would be from the maps tablenames list

resource "aws_glue_catalog_table" "my_glue_table" {
  for_each      = var.my_map    
  name          = "xxxxxxxxxxxxx" #--> here i needed the name to be from the tablename of my_map
  database_name = aws_glue_catalog_database.my_glue_db[each.key].name
  table_type    = "EXTERNAL_TABLE"

  partition_keys {
    name = "date"
    type = "date"
  }

  storage_descriptor {
    columns {
      name = "file_name"
      type = "string"
    }
    .
    .
    .

    compressed    = false
    location      = "s3://${aws_s3_bucket.my_bucket[each.key].id}/suffix/" ##### here suffix would be from the maps tablenames list
    input_format  = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"
    output_format = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"

    ser_de_info {
      name                  = "ParquetHiveSerDe"
      serialization_library = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
    }
  }
}

I need to create 2 glue DB, s3 bucket and 4 glue tables dependent on these 2 resources

Upvotes: 0

Views: 1840

Answers (1)

Marcin
Marcin

Reputation: 238727

Is it possible to use for_each inside a for_each

Yes. This will result in a flat version of your my_map, which you can easily iterate over:

locals {
  flat_my_map = merge([
      for k,v in var.my_map: {
        for tname in v.tablenames: 
          "${k}-${tname}" => {
            key1 = k
            name = v.name
            tablename = tname
          }
      }
  ]...) # do NOT remove the dots
}

then

resource "aws_glue_catalog_table" "my_glue_table" {

  for_each      = var.flat_my_map    

  name          = each.value.tablename
  database_name = aws_glue_catalog_database.my_glue_db[each.value.key1].name
  table_type    = "EXTERNAL_TABLE"

  partition_keys {
    name = "date"
    type = "date"
  }

  storage_descriptor {
    columns {
      name = "file_name"
      type = "string"
    }
    .
    .
    .

    compressed    = false
    location      = "s3://${aws_s3_bucket.my_bucket[each.value.key1].id}/suffix/${each.value.name}" ##### here suffix would be from the maps tablenames list
    input_format  = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"
    output_format = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"

    ser_de_info {
      name                  = "ParquetHiveSerDe"
      serialization_library = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
    }
  }
}

Upvotes: 1

Related Questions