Terraform AWS datasource aws_subnet returns no matching subnets found

Question

I have a module that I'm attempting to find AWS subnets within and then use/return. It's called like this:

module "smurf_subnet_grp" {
   source            = "../../modules/networking/subnet_grp_per_az-test"
   vpc_id            = "${module.networking_uswe2.vpc_id}"
   azs               = "${local.az_list_uswe2}"
   private_subnets   = "${var.private_subnets_uswe2}"
}

Code for the module:

variable "azs"              { type = "list" }
variable "private_subnets"  { type = "list" }
variable "vpc_id"           {}

# ========== remove special subnets ==============

locals {
  cnt      = "${length(var.private_subnets) - 3}"
  prv_subs = "${slice(var.private_subnets, 0, local.cnt)}"
}

# ========== get subnet details ==================

data "aws_subnet" "self" {
  count         = "${length(local.prv_subs)}"
  vpc_id        = "${var.vpc_id}"
  cidr_block    = "${local.prv_subs[count.index]}"
}

# ========== get subnets by AZ ===================

locals {
  prv_subs0 = "${matchkeys(data.aws_subnet.self.*.id, data.aws_subnet.self.*.availability_zone, list(var.azs[0]))}"
  prv_subs1 = "${matchkeys(data.aws_subnet.self.*.id, data.aws_subnet.self.*.availability_zone, list(var.azs[1]))}"
  prv_subs2 = "${matchkeys(data.aws_subnet.self.*.id, data.aws_subnet.self.*.availability_zone, list(var.azs[2]))}"
}

# ========== select 1 subnet per AZ ==============

resource "random_shuffle" "prv_sub0" {
    input        = ["${local.prv_subs0}"]
    result_count = 1
}
resource "random_shuffle" "prv_sub1" {
    input        = ["${local.prv_subs1}"]
    result_count = 1
}
resource "random_shuffle" "prv_sub2" {
    input        = ["${local.prv_subs2}"]
    result_count = 1
}

# ========== put selected into 1 list ============

locals {
  prv_sub_az = [
       "${random_shuffle.prv_sub0.result}", 
       "${random_shuffle.prv_sub1.result}", 
       "${random_shuffle.prv_sub2.result}"
  ]
}

output "prv_subnet_grp" {
  value = "${local.prv_sub_az}"
}

Which throws this:

Error: Error refreshing state: 1 error occurred:
    * module.smurf_subnet_grp.data.aws_subnet.self: 6 errors occurred:
    * module.smurf_subnet_grp.data.aws_subnet.self[5]: data.aws_subnet.self.5: no matching subnet found
    * module.smurf_subnet_grp.data.aws_subnet.self[3]: data.aws_subnet.self.3: no matching subnet found
    * module.smurf_subnet_grp.data.aws_subnet.self[0]: data.aws_subnet.self.0: no matching subnet found
    * module.smurf_subnet_grp.data.aws_subnet.self[1]: data.aws_subnet.self.1: no matching subnet found
    * module.smurf_subnet_grp.data.aws_subnet.self[2]: data.aws_subnet.self.2: no matching subnet found
    * module.smurf_subnet_grp.data.aws_subnet.self[4]: data.aws_subnet.self.4: no matching subnet found

If I introduce a depends_on for the aws_subnet data provider:

data "aws_subnet" "self" {
  count         = "${length(local.prv_subs)}"
  vpc_id        = "${var.vpc_id}"
  cidr_block    = "${local.prv_subs[count.index]}"
  depends_on    = ["null_resource.module_depends_on"]
}

It'll work as expected but then will recreate it every time.

UPDATE #1

In an attempt to work around this I tried implementing a suggestion found here on the Hashicorp discussion forums titled: TIPS: Howto implement Module depends_on emulation. The theory being that the issue I was running into was an order/dependency issue.

The code I'm using to implement depends_on is below for my subnet_grp_per_az-test module:

/*
    Add the following line to the resource in this module that depends on the completion of external module components:

    depends_on = ["null_resource.module_depends_on"]

    This will force Terraform to wait until the dependant external resources are created before proceeding with the creation of the
    resource that contains the line above.

    This is a hack until Terraform officially support module depends_on.
*/

variable "module_depends_on" {
  default = [""]
}

resource "null_resource" "module_depends_on" {
  triggers = {
    value = "${length(var.module_depends_on)}"
  }
}

Answer 1

Note that this is my understanding. Hopefully M.Atkins can confirm.

Why Terraform has no depends_on for module

• Module Cannot be Used in depends_on #18239

As a preparation, first would like to clarify a misunderstanding we may have (which I had) about Terraform module.

The objective is establish that there is no dependency from a TF module to another TF module. Just because module A declaration comes before module B as in the Root module tf file below does not mean the creation of resource in module B will not happen until the resources in module A all complete.

What would happen if we have two modules A and B and they depend on each other?

Root module

resource "aws_vpc" "this" {
  cidr_block = var.vpc_cidr
}
#--------------------------------------------------------------------------------
# Create PRIVATE subnets but create EC2 in PUBLIC subnets (cross module reference)
#--------------------------------------------------------------------------------
module "private_subnet_public_ec2" {
  source               = "../private_subnet_public_ec2"
  vpc_id               = aws_vpc.this.id
  private_subnet_cidr_blocks = var.private_subnet_cidr_blocks
  public_subnet_ids    = module.public_subnet_private_ec2.public_subnet_ids
  ami_id               = data.aws_ami.this.id
}
#--------------------------------------------------------------------------------
# Create PUBLIC subnets but create EC2 in PRIVATE subnets (cross module reference)
#--------------------------------------------------------------------------------
module "public_subnet_private_ec2" {
  source              = "../public_subnet_private_ec2"
  vpc_id              = aws_vpc.this.id
  public_subnet_cidr_blocks = var.public_subnet_cidr_blocks
  private_subnet_ids  = module.private_subnet_public_ec2.private_subnet_ids
  ami_id              = data.aws_ami.this.id
}

Module A

Creates private subnets and EC2s in the public subnets created in module B.

resource "aws_subnet" "private" {
  count = length(var.private_subnet_cidr_blocks)
  vpc_id     = var.vpc_id
  cidr_block = var.private_subnet_cidr_blocks[count.index]
}
output "private_subnet_ids" {
  value = aws_subnet.private[*].id
}

resource "aws_instance" "public_ec2" {
  count = length(var.public_subnet_ids)
  subnet_id = var.public_subnet_ids[count.index]
  ami           = var.ami_id
  instance_type = "t2.micro"
  tags = {
    Name = "PublicEC2${count.index}}"
  }
  provisioner "local-exec" {
    command = <<EOF
echo "Public EC2 ${count.index} ID is ${self.id}"
EOF
  }
}

Module B

Creates pubic subnets and EC2s in the private subnets created in module A.

resource "aws_subnet" "public" {
  count = length(var.public_subnet_cidr_blocks)
  vpc_id     = var.vpc_id
  cidr_block = var.public_subnet_cidr_blocks[count.index]
}
output "public_subnet_ids" {
  value = aws_subnet.public[*].id
}

resource "aws_instance" "private_ec2" {
  count = length(var.private_subnet_ids)
  subnet_id = var.private_subnet_ids[count.index]
  ami           = var.ami_id
  instance_type = "t2.micro"
  tags = {
    Name = "privateEC2${count.index}}"
  }
  provisioner "local-exec" {
    command = <<EOF
echo "private EC2 ${count.index} ID is ${self.id}"
EOF
  }
}

The execution result:

$ terraform apply --auto-approve
Apply complete! Resources: 13 added, 0 changed, 0 destroyed.

So what we need to have in mind is what Terraform actually sees, which is a flatten world without modules where only resources exist. Terraform creates a DAG from the resources and there is no module as a node in the DAG.

This is why we cannot use depends_on against a module because in the Terraform DAG , a module is not a node to which vertices can be created to represent dependencies.

Synchronization Monitor Mechanism

Terraform executes multiple threads to create resources in a concurrent manner. We need a synchronization monitor on which threads can wait so that those threads start only when the dependency resource has been created.

The monitor mechanism in Terraform (other than HCL depends_on statement) is using the attribute(s) of the resource created (or reference via local).

In the TIPS: Howto implement Module depends_on emulation, Martin Atkins showed a example using a module variable "vm_depends_on" referring to the attribute of the firewall resource created, module.fw_core01.firewall so that those resources in the module "example" can only be created after the firewall has been created.

---

Problem

In my understanding, the cause of the problem in this question is the lack of monitor, a reference to an attribute of the resource that should have been already created when a thread started executing data "aws_subnet" "self".

As I do not have the entire source of the original question, here is an example to reproduce the error.

Root module

variable "vpc_cidr" {
    default = "10.5.0.0/20"
}
variable "private_subnet_cidr_blocks" {
  default = ["10.5.3.0/24","10.5.4.0/24","10.5.5.0/24"]
}

resource "aws_vpc" "this" {
  cidr_block = var.vpc_cidr
}
module "private_subnet" {
  source          = "../private_subnet"
  vpc_id          = aws_vpc.this.id
  private_subnet_cidr_blocks = var.private_subnet_cidr_blocks
}
module "private_ec2" {
  source          = "../private_ec2"
  vpc_id          = aws_vpc.this.id
  private_subnet_cidr_blocks = var.private_subnet_cidr_blocks
  ami_id          = data.aws_ami.this.id
}

Module Subnet

variable "vpc_id" {}
variable "private_subnet_cidr_blocks" {
  type = list(string)
}

resource "aws_subnet" "private" {
  count = length(var.private_subnet_cidr_blocks)
  vpc_id     = var.vpc_id
  cidr_block = var.private_subnet_cidr_blocks[count.index]
}

Module EC2

variable "vpc_id" {}
variable "private_subnet_cidr_blocks" {
  type = list(string)
}
variable "ami_id" {}

data aws_subnet "private" {
  count      = length(var.private_subnet_cidr_blocks)
  vpc_id     = var.vpc_id
  cidr_block = var.private_subnet_cidr_blocks[count.index]
}

resource "aws_instance" "private_ec2" {
  count         = length(data.aws_subnet.private[*].id)
  subnet_id     = data.aws_subnet.private[count.index].id
  ami           = var.ami_id
  instance_type = "t2.micro"
  tags          = {
    Name = "privateEC2${count.index}}"
  }
  provisioner "local-exec" {
    command = <<EOF
echo "private EC2 ${count.index} ID is ${self.id}"
EOF
  }
}

The execution result:

$ terraform apply --auto-approve
...
Error: no matching subnet found

  on ../private_ec2/main.tf line 1, in data "aws_subnet" "private":
   1: data aws_subnet "private" {

---

Cause

The cause is passing the private subnet CIDR using constant variables, NOT the attributes of the AWS subnet created.

If the attribute(s) of the AWS subnet resource created is used, then it works as the monitor on which the thread that executes data "aws_subnet" "self" will wait on.

Root module

variable "private_subnet_cidr_blocks" {
  default = ["10.5.3.0/24","10.5.4.0/24","10.5.5.0/24"]
}
module "private_ec2" {
  source          = "../private_ec2"
  vpc_id          = aws_vpc.this.id
  private_subnet_cidr_blocks = var.private_subnet_cidr_blocks # <----- Here
  ami_id          = data.aws_ami.this.id
}

Because there is no dependency between the resource in Module Subnet and that in Module EC2, the resource creation in both modules run in parallel.

In the original question

I believe, the cause of the original question is the private_subnets passing variable, not the attributes of the actually created AWS subnet resource(s).

module "smurf_subnet_grp" {
   source            = "../../modules/networking/subnet_grp_per_az-test"
   vpc_id            = "${module.networking_uswe2.vpc_id}"
   azs               = "${local.az_list_uswe2}"

   private_subnets   = "${var.private_subnets_uswe2}" <----- Here

}

Therefore, data "aws_subnet" "self" is executed concurrently while the AWS subnet is being or yet to be created.

---

Fix

The fix for the example:

Root module

resource "aws_vpc" "this" {
  cidr_block = var.vpc_cidr
}
module "private_subnet" {
  source          = "../private_subnet"
  vpc_id          = aws_vpc.this.id
  private_subnet_cidr_blocks = var.private_subnet_cidr_blocks
}
module "private_ec2" {
  source          = "../private_ec2"
  vpc_id          = aws_vpc.this.id

  #--------------------------------------------------------------------------------
  # Pass the attributes of created aws_subnet resource attributes
  #--------------------------------------------------------------------------------
  #private_subnet_cidr_blocks = var.private_subnet_cidr_blocks
  private_subnet_cidr_blocks = module.private_subnet.private_subnet_cidr_blocks  # <--- Here
  #--------------------------------------------------------------------------------

  ami_id          = data.aws_ami.this.id
}

Module subnet

resource "aws_subnet" "private" {
  count = length(var.private_subnet_cidr_blocks)
  vpc_id     = var.vpc_id
  cidr_block = var.private_subnet_cidr_blocks[count.index]
}
#--------------------------------------------------------------------------------
# Output the cidr_block attributes of the AWS subnet resources created
#--------------------------------------------------------------------------------
output "private_subnet_cidr_blocks" {
  value = aws_subnet.private[*].cidr_block    # <----- Here
}

Result

$ terraform apply --auto-approve
data.aws_availability_zones.all: Refreshing state...
aws_vpc.this: Refreshing state... [id=vpc-0b338898d18a5986e]
data.aws_ami.this: Refreshing state...
data.aws_region.current: Refreshing state...
module.private_subnet.data.aws_ami.ubuntu: Refreshing state...
module.private_subnet.aws_subnet.private[2]: Refreshing state... [id=subnet-0cf916b6b9003f71f]
module.private_subnet.aws_subnet.private[1]: Refreshing state... [id=subnet-0b39beb22b23eef5d]
module.private_subnet.aws_subnet.private[0]: Refreshing state... [id=subnet-0c80c92f4023ba893]
aws_vpc.this: Creating...
aws_vpc.this: Still creating... [10s elapsed]
aws_vpc.this: Still creating... [20s elapsed]
aws_vpc.this: Creation complete after 24s [id=vpc-00069d144b5f76182]
module.private_subnet.aws_subnet.private[1]: Creating...
module.private_subnet.aws_subnet.private[2]: Creating...
module.private_subnet.aws_subnet.private[0]: Creating...
module.private_subnet.aws_subnet.private[2]: Creation complete after 5s [id=subnet-0252c6047cd56abac]
module.private_subnet.aws_subnet.private[1]: Creation complete after 6s [id=subnet-019f8cbd30db10edb]
module.private_subnet.aws_subnet.private[0]: Creation complete after 6s [id=subnet-0a1028bf17d7d81be]
module.private_ec2.data.aws_subnet.private[1]: Refreshing state...
module.private_ec2.data.aws_subnet.private[2]: Refreshing state...
module.private_ec2.data.aws_subnet.private[0]: Refreshing state...
module.private_ec2.aws_instance.private_ec2[2]: Creating...
module.private_ec2.aws_instance.private_ec2[1]: Creating...
module.private_ec2.aws_instance.private_ec2[0]: Creating...
module.private_ec2.aws_instance.private_ec2[2]: Still creating... [10s elapsed]
module.private_ec2.aws_instance.private_ec2[1]: Still creating... [10s elapsed]
module.private_ec2.aws_instance.private_ec2[0]: Still creating... [10s elapsed]
module.private_ec2.aws_instance.private_ec2[2]: Still creating... [20s elapsed]
module.private_ec2.aws_instance.private_ec2[1]: Still creating... [20s elapsed]
module.private_ec2.aws_instance.private_ec2[0]: Still creating... [20s elapsed]
module.private_ec2.aws_instance.private_ec2[2]: Still creating... [30s elapsed]
module.private_ec2.aws_instance.private_ec2[1]: Still creating... [30s elapsed]
module.private_ec2.aws_instance.private_ec2[0]: Still creating... [30s elapsed]
module.private_ec2.aws_instance.private_ec2[2]: Still creating... [40s elapsed]
module.private_ec2.aws_instance.private_ec2[1]: Still creating... [40s elapsed]
module.private_ec2.aws_instance.private_ec2[0]: Still creating... [40s elapsed]
module.private_ec2.aws_instance.private_ec2[1]: Provisioning with 'local-exec'...
module.private_ec2.aws_instance.private_ec2[1] (local-exec): Executing: ["/bin/sh" "-c" "echo \"private EC2 1 ID is i-0ced265565dfec85c\"\n"]
module.private_ec2.aws_instance.private_ec2[1] (local-exec): private EC2 1 ID is i-0ced265565dfec85c
module.private_ec2.aws_instance.private_ec2[1]: Creation complete after 46s [id=i-0ced265565dfec85c]
module.private_ec2.aws_instance.private_ec2[0]: Provisioning with 'local-exec'...
module.private_ec2.aws_instance.private_ec2[0] (local-exec): Executing: ["/bin/sh" "-c" "echo \"private EC2 0 ID is i-0f6ce62c29376c6fe\"\n"]
module.private_ec2.aws_instance.private_ec2[0] (local-exec): private EC2 0 ID is i-0f6ce62c29376c6fe
module.private_ec2.aws_instance.private_ec2[0]: Creation complete after 47s [id=i-0f6ce62c29376c6fe]
module.private_ec2.aws_instance.private_ec2[2]: Provisioning with 'local-exec'...
module.private_ec2.aws_instance.private_ec2[2] (local-exec): Executing: ["/bin/sh" "-c" "echo \"private EC2 2 ID is i-03be32b7b803eb0cc\"\n"]
module.private_ec2.aws_instance.private_ec2[2] (local-exec): private EC2 2 ID is i-03be32b7b803eb0cc
module.private_ec2.aws_instance.private_ec2[2]: Creation complete after 50s [id=i-03be32b7b803eb0cc]

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

---

Fix for the original question

I believe below would fix the issue.

module "smurf_subnet_grp" {
   source            = "../../modules/networking/subnet_grp_per_az-test"
   vpc_id            = "${module.networking_uswe2.vpc_id}"
   azs               = "${local.az_list_uswe2}"

   #--------------------------------------------------------------------------------
   # Pass the cidr_block attribute of aws_subnet resource created in module.networking_uswe2
   #------------------------------------------------------------------------------   
   #private_subnets   = "${var.private_subnets_uswe2}"
   private_subnets   = module.networking_uswe2.private_subnet_cidr_blocks  # <---- Here
   #------------------------------------------------------------------------------   
}