Aws_acm_certificate.app_cert.domain_validation_options is a set of object, known only after apply

Question

I've been working through a Terraform (+ CI/CD) workshop which was taught in an earlier version of Terraform, but I decided to it in 1.0.11 with AWS provider 3.65.0.... just to see what sort of the difference would be. I've hit a blocker when dealing with ACM to get the certificate, and I need some advice on how to proceed.

The error I'm getting is during the plan stage:

[ckerr@ck-vm-rhel8-localdomain recipe-app-api-devops]$ docker-compose -f deploy/docker-compose.yml run --rm terraform plan
Creating deploy_terraform_run ... done
╷
│ Error: Invalid for_each argument
│ 
│   on dns.tf line 44, in resource "aws_route53_record" "app_cert_validation_records":
│   44:   for_each = {
│   45:     for dvo in aws_acm_certificate.app_cert.domain_validation_options : dvo.domain_name => {
│   46:       name   = dvo.resource_record_name
│   47:       type   = dvo.resource_record_type
│   48:       record = dvo.resource_record_value
│   49:     }
│   50:   }
│     ├────────────────
│     │ aws_acm_certificate.app_cert.domain_validation_options is a set of object, known only after apply
│ 
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how
│ many instances will be created. To work around this, use the -target argument to first apply only the resources that the
│ for_each depends on.
╵
Releasing state lock. This may take a few moments...
ERROR: 1

I have a file called dns.tf that looks after the DNS and SSL certificates, with the intention that Amazon's Certificate Manager (ACM) will acquire the certificate. That file looks like this, and I've included commentary in the hope that someone might be able to identify where my thinking has gone wrong.

// BELIEF: this is the DNS zone we will be operating in. No issues there.
//
data "aws_route53_zone" "zone" {
  name = "${var.dns_zone_name}."
}

// BELIEF: this creates a CNAME record that points to our AWS ELB instance for our application.
// This is where it departs a little from the documentation, but I'm not sure if the documentation
// is just being a bit sparse, or if the validation records are meant to also be part of this object.
//
resource "aws_route53_record" "app" {
  zone_id = data.aws_route53_zone.zone.zone_id
  name    = "${lookup(var.subdomain, terraform.workspace)}.${data.aws_route53_zone.zone.name}"
  type    = "CNAME"
  ttl     = "300"

  records = [aws_lb.api.dns_name]
}

// BELIEF: this models the certificate for our application.
//
resource "aws_acm_certificate" "app_cert" {
  domain_name       = aws_route53_record.app.fqdn
  validation_method = "DNS"

  tags = local.common_tags

  lifecycle {
    create_before_destroy = true
  }
}

// BELIEF: to prove ownership of the domain we need to be able to prove that we can
// place certain records into the domain as part of a DNS challenge method.
// I believe this is meant to refer to just those challenge records.
// As this is the dynamic part, its unlikely that the domain validation options would
// be known ahead of time, and so this is where I'm coming unstuck.
//
resource "aws_route53_record" "app_cert_validation_records" {
  allow_overwrite = true
  zone_id         = data.aws_route53_zone.zone.zone_id
  ttl             = "60"

  for_each = {
    for dvo in aws_acm_certificate.app_cert.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      type   = dvo.resource_record_type
      record = dvo.resource_record_value
    }
  }

  name    = each.value.name
  type    = each.value.type
  records = [each.value.record]
}

// BELIEF: This doesn't create anything; its just a placeholder for the validation
// process... presumably for dependency reasons.
// It basically just associates the FQDN (in the certificate) with its validation records.
//
resource "aws_acm_certificate_validation" "app_cert_validation_process" {
  certificate_arn         = aws_acm_certificate.app_cert.arn
  validation_record_fqdns = [for record in aws_route53_record.app_cert_validation_records : record.fqdn]
}

// Reading https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-3-upgrade#resource-aws_acm_certificate
// I don't see where the problem is.

In case it matters, there is another resource that is depending on this, which is the ELB instance which uses the certificate:

resource "aws_lb_listener" "api_https" {
  load_balancer_arn = aws_lb.api.arn
  port              = 443
  protocol          = "HTTPS"
  certificate_arn   = aws_acm_certificate_validation.app_cert_validation_process.certificate_arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.api.arn
  }
}

I've been looking to follow https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-3-upgrade#resource-aws_acm_certificate, which indicates that a plan should work and I should see the text "(known after apply)" in the output.

I take it, based on my research to date, is that the root cause is that the AWS provider can't / won't predict what the domain validation options will be ahead of time.... and yet the upgrade documentation indicates that, while this is only something that is known after applying, it shouldn't cause a plan failure and I shouldn't need any ugly -target workaround.

I tried using a depends_on, but that's not going to help with the plan stage.

Full code is in https://gitlab.com/cameron.kerr.nz/recipe-app-api-devops/ if it helps.

Thanks for reading, Cameron

PS. I've asked this previously in https://discuss.hashicorp.com/t/aws-acm-certificate-app-cert-domain-validation-options-is-a-set-of-object-known-only-after-apply/31952, but so far no reply.

Answer 1

TL;DR: This fixes your example from my testing:

 resource "aws_acm_certificate" "app_cert" {
-  domain_name       = aws_route53_record.app.fqdn
+  domain_name       = aws_route53_record.app.name
   validation_method = "DNS"

But not so fast, this only works because for the record you are creating in aws_route53_record.app has the same name as the final fqdn.

As per the docs for the aws_route53_record resource, fqdn is built using the zone domain and the name parameter.

For further explanation, in the working example, we can see that fqdn is not known, but name is. Which the for_each in aws_route53_record.app_cert_validation_records doesn't like.

  # aws_route53_record.app will be created
  + resource "aws_route53_record" "app" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "foo.example.com"
      + records         = [
          + "xxxx.alb.amazon.com",
        ]
      + ttl             = 300
      + type            = "CNAME"
      + zone_id         = "XXXXXXXXXXX"
    }