terraform 0.13.5 resources overwrite each other on consecutive calls

Question

• I am using terraform 0.13.5 to create aws_iam resources

• I have 2 terraform resources as follows

  module "calls_aws_iam_policy_attachment" {
    # This calls an external module to 
    # which among other things creates a policy attachment 
    # resource attaching the roles to the policy
    source   = ""
    name = "xoyo"
    roles = ["rolex", "roley"]
    policy_arn = "POLICY_NAME"
  }
  
  resource "aws_iam_policy_attachment" "policies_attached" {
    # This creates a policy attachment resource attaching the roles to the policy
    # The roles here are a superset of the roles in the above module
    roles = ["role1", "role2", "rolex", "roley"]
    policy_arn = "POLICY_NAME"
    name = "NAME"
    # I was hoping that adding the depends on block here would mean this 
    # resource is always created after the above module
    depends_on = [ module.calls_aws_iam_policy_attachment ]
  }
  

• The first module creates a policy and attaches some roles. I cannot edit this module

• The second resource attaches more roles to the same policy along with other policies

• the second resource depends_on the first resource, so I would expect that the policy attachements of the second resource always overwrite those of the first resource

• In reality, the policy attachments in each resource overwrite each other on each consecutive build. So that on the first build, the second resources attachments are applied and on the second build the first resources attachements are applied and so on and so forth.

• Can someone tell me why this is happening? Does depends_on not work for resources that overwrite each other?

• Is there an easy fix without combining both my resources together into the same resource?

Answer 1

As to why this is happening:

• during the first run terraform deploys the first resources, then the second ones - this order is due to the depends_on relation (the next steps work regardless of any depends_on). The second ones overwrite the first ones
• during the second deploy terraform looks at what needs to be done:
  • the first ones are missing (were overwritten), they need to be created
  • the second ones are fine, terraform ignores them for this update
  • now only the first ones will be created and they will overwrite the second ones
• during the third run the same happens but the exact other way around, seconds are missing, first are ignored, second overwrite first
• repeat as often as you want, you will never end up with a stable deployment.

Solution: do not specify conflicting things in terraform. Terraform is supposed to be a description of what the infrastructure should look like - and saying "this resource should only have property A" and "this resource should only have property B" is contradictory, terraform will not be able to handle this gracefully.

What you should do specifically: do not use aws_iam_policy_attachment, basically ever, look at the big red box in the docs. Use multiple aws_iam_role_policy_attachment instead, they are additive, they will not overwrite each other.