Terraform resource for_each with nested dynamic block keeps re-applying the same changes

Question

So I have created a google_bigquery module to create datasets and set access.

The module iterates over a map of list of maps. It uses the each.key to create the datasets then iterates over the list of maps to create the dynamic access.

The module works as in:

• It has no errors nor warning
• It deploys the resources
• It populates the remote statefile appropriately.

The issue is that everytime I ran terraform it wants to re-apply the same changes, over and over again.

Clearly something is not right but not sure what.

here is the code

main.tf

locals {
  env           = basename(path.cwd)
  project       = basename(abspath("${path.cwd}/../.."))
  project_name  = coalesce(var.project_name, format("%s-%s", local.project, local.env))
}

data "google_compute_zones" "available" {
  project = local.project_name
  region  = var.region
}

provider "google" {
  project = local.project_name
  region  = var.region
  version = "~> 2.0" #until 3.0 goes out of beta
}

terraform {
  required_version = ">= 0.12.12"
}

resource "google_bigquery_dataset" "main" {
  for_each                   = var.datasets
  dataset_id                 = upper("${each.key}_${local.env}")
  location                   = var.region
  delete_contents_on_destroy = true

  dynamic "access" {
    for_each = flatten([ for k, v in var.datasets : [
                 for i in each.value : {
                   role           = i.role
                   user_by_email  = i.user_by_email
                   group_by_email = i.group_by_email
                   dataset_id     = i.dataset_id
                   project_id     = i.project_id
                   table_id       = i.table_id
    }]])
    content {
      role           = lookup(access.value,"role", "")
      user_by_email  = lookup(access.value,"user_by_email","")
      group_by_email = lookup(access.value,"group_by_email","")
      view {
        dataset_id   = lookup(access.value,"dataset_id","")
        project_id   = lookup(access.value,"project_id","")
        table_id     = lookup(access.value,"table_id", "")
        }
    }
  }



  access {
    role          = "READER"
    special_group = "projectReaders"
  }

  access {
    role           = "OWNER"
    group_by_email = "Group"
  }

  access {
    role           = "OWNER"
    user_by_email  = "ServiceAccount"
  }

  access {
    role          = "WRITER"
    special_group = "projectWriters"
  }

}

variables.tf

variable "region" {
  description = ""
  default     = ""
}

variable "env" {
  default = ""
}

variable "project_name" {
  default = ""
}

variable "owner_group" {
  description = ""
  default     = ""
}

variable "owner_sa" {
  description = ""
  default = ""
}

variable "datasets" {
  description = "A map of objects, including dataset_isd abd access"
  type = map(list(map(string)))
}

terraform.tfvars

datasets = {
  dataset01 = [
    {
      role           = "WRITER"
      user_by_email  = "email_address"
      group_by_email = ""
      dataset_id     = ""
      project_id     = ""
      table_id       = ""
    },
    {
      role           = ""
      user_by_email  = ""
      group_by_email = ""
      dataset_id     ="MY_OTHER_DATASET"
      project_id     ="my_other_project"
      table_id       ="my_test_view"
    }
  ]
  dataset02 = [
    {
      role           = "READER"
      user_by_email  = ""
      group_by_email = "group"
      dataset_id     = ""
      project_id     = ""
      table_id       = ""
    },
    {
      role           = ""
      user_by_email  = ""
      group_by_email = ""
      dataset_id     ="MY_OTHER_DATASET"
      project_id     ="my_other_project"
      table_id       ="my_test_view_2"
    }
  ]
}

So the problem is that the dynamic block (the way I wrote it) can generate this output

      + access {
          + role          = "WRITER"
          + special_group = "projectWriters"

          + view {}
        }

this is applied, no errors, but it will want to re-apply it over and over

The issue seems to be that the provider API response doesn't include the empty view{}

Any suggestion how I could make the view block conditional on the values of it being not null?

Answer 1

I fixed the problem. I changed the module slightly and the variable type.

I have split the roles and the views into their own lists of maps within the parent map of datasets.

There are conditionals in each block so the dynamic block is only applied if the roles exists or views exists.

Also realized the dynamic block was iterating on the wrong iterator.

The dynamic block was iterating on var.datasets which was causing the permissions assigned to each dataset to be applied to all datasets. So now it has been changed to iterate on each.value (from the resource for_each).

Here is the new code that works

MAIN.TF

resource "google_bigquery_dataset" "main" {
  for_each                   = var.datasets
  dataset_id                 = upper("${each.key}_${local.env}")
  location                   = var.region
  delete_contents_on_destroy = true

  dynamic "access" {
    for_each = flatten([for i in each.value : [
      for k, v in i : [
        for l in v :
        {
          role           = l.role
          user_by_email  = l.user_by_email
          group_by_email = l.group_by_email
          special_group  = l.special_group
      }]
      if k == "roles"
    ]])
    content {
      role           = access.value["role"]
      user_by_email  = access.value["user_by_email"]
      group_by_email = access.value["group_by_email"]
      special_group  = access.value["special_group"]
    }
  }

  dynamic "access" {
    for_each = flatten([for i in each.value : [
      for k, v in i : [
        for l in v :
        {
          dataset_id = l.dataset_id
          project_id = l.project_id
          table_id   = l.table_id
      }]
      if k == "views"
    ]])
    content {
      view {
        dataset_id = access.value["dataset_id"]
        project_id = access.value["project_id"]
        table_id   = access.value["table_id"]
      }
    }
  }
}

VARIABLES.TF

variable "datasets" {
  description = "A map of objects, including datasets IDs, roles and views"
  type        = map(list(map(list(map(string)))))
  default     = {}
}

continued....

Terraform.tfvars

datasets = {
  dataset01 = [
    {
      roles = [
        {
          role="WRITER"
          user_by_email="email_address"
          group_by_email=""
          special_group=""
        }
      ]
    views = [
        {
          dataset_id="MY_OTHER_DATASET"
          project_id="my_other_project"
          table_id="my_test_view"
        }
      ]
    }
  ]
  dataset02 = [
    {
      roles = [
        {
          role="READER"
          user_by_email=""
          group_by_email="group"
          special_group=""
        }
      ]
      views=[
        {
          dataset_id="MY_OTHER_DATASET"
          project_id="my_other_project"
          table_id="my_test_view_2"
        }
      ]
    }     
  ]
}