Cannot create an Elastic Beanstalk application in a custom VPC using Terraform

Question

I am trying to create an Elastic Beanstalk application in a private subnet of a custom VPC using Terraform. The problem I have is that the creation hits a time-out.

Error:

Error waiting for Elastic Beanstalk Environment (e-xxxxxxxxx) to become ready: 2 errors occurred:

• 2021-01-22 16:37:56.664 +0000 UTC (e-xxxxxxxxx) : Stack named 'awseb-e-xxxxxxxxx-stack' aborted operation. Current state: 'CREATE_FAILED' Reason: The following resource(s) failed to create: [AWSEBInstanceLaunchWaitCondition].
• 2021-01-22 16:37:56.791 +0000 UTC (e-xxxxxxxxx) : The EC2 instances failed to communicate with AWS Elastic Beanstalk, either because of configuration problems with the VPC or a failed EC2 instance. Check your VPC configuration and try launching the environment again.

I think I am missing a connection from the VPC to the outside, but I'm not sure what.

I have tried to add some aws_vpc_endpoint resources:

resource "aws_vpc" "main" {
  cidr_block = "10.16.0.0/16"

  enable_dns_support   = true
  enable_dns_hostnames = true
}

resource "aws_security_group" "elasticbeanstalk_vpc_endpoint" {
  name   = "elasticbeanstalk-vpc-endpoint"
  vpc_id = aws_vpc.main.id

  ingress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = [ "0.0.0.0/0" ]
  }

  egress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = [ "0.0.0.0/0" ]
  }
}

resource "aws_vpc_endpoint" "elasticbeanstalk" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.${var.aws_region}.elasticbeanstalk"

  security_group_ids = [
    aws_security_group.elasticbeanstalk_vpc_endpoint.id,
  ]

  vpc_endpoint_type = "Interface"
}

resource "aws_security_group" "ec2_vpc_endpoint" {
  name   = "ec2-vpc-endpoint"
  vpc_id = aws_vpc.main.id

  ingress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = [ "0.0.0.0/0" ]
  }

  egress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = [ "0.0.0.0/0" ]
  }
}

resource "aws_vpc_endpoint" "ec2" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.${var.aws_region}.ec2"

  security_group_ids = [
    aws_security_group.ec2_vpc_endpoint.id,
  ]

  vpc_endpoint_type = "Interface"
}

EB instance:

resource "aws_elastic_beanstalk_environment" "endpoint" {
  name                = "foo-env"

  setting {
    namespace = "aws:ec2:vpc"
    name      = "VPCId"
    value     = aws_vpc.main.vpc_id
    resource  = ""
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "Subnets"
    value     = join(",", [ aws_subnet.private_a.id, aws_subnet.private_b.id ])
    resource  = ""
  }

  setting {
    namespace = "aws:ec2:vpc"
    name      = "ELBScheme"
    value     = "internal"
    resource  = ""
  }

  setting {
    namespace = "aws:elasticbeanstalk:environment"
    name      = "EnvironmentType"
    value     = "LoadBalanced"
    resource  = ""
  }

  setting {
    namespace = "aws:elasticbeanstalk:environment"
    name      = "LoadBalancerType"
    value     = "application"
    resource  = ""
  }

  setting {
    namespace = "aws:elasticbeanstalk:application:environment"
    name      = "AWS_REGION"
    value     = var.aws_region
    resource  = ""
  }

  # ...

  depends_on = [
    aws_vpc_endpoint.ec2,
    aws_vpc_endpoint.elasticbeanstalk,
  ]
}

---

Subnets created like this:

resource "aws_subnet" "public" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "10.16.0.0/24"

  availability_zone       = "${var.aws_region}c"
  map_public_ip_on_launch = true
}

resource "aws_subnet" "private_a" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "10.16.192.0/24"

  availability_zone = "${var.aws_region}a"
}

resource "aws_subnet" "private_b" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "10.16.224.0/24"

  availability_zone = "${var.aws_region}b"
}

Route tables:

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id
}

resource "aws_route_table_association" "private_a" {
  route_table_id = aws_route_table.private.id
  subnet_id      = aws_subnet.private_a.id
}

resource "aws_route_table_association" "private_b" {
  route_table_id = aws_route_table.private.id
  subnet_id      = aws_subnet.private_b.id
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
}

resource "aws_route_table_association" "public" {
  route_table_id = aws_route_table.public.id
  subnet_id      = aws_subnet.public.id
}

resource "aws_internet_gateway" "public" {
  vpc_id = aws_vpc.main.id
}

resource "aws_route" "public_internet" {
  route_table_id = aws_route_table.public.id
  gateway_id     = aws_internet_gateway.public.id

  destination_cidr_block = "0.0.0.0/0"
}

Answer 1

Some observations which can contribute to your issue are listed below. General guide on setting up EB in private VPC is in aws docs.

The observations:

• aws_vpc.main.vpc_id should be aws_vpc.main.id.

• in all your endpoints private_dns_enabled is false by default. It should be true for the endpoints to work seamlessly. You can add the following to your endpoints:

private_dns_enabled = true

• your endpoints are not associated with any subnets. To associate them you can use:

subnet_ids        = [aws_subnet.private_a.id, aws_subnet.private_b.id]

• there is no interface endpoint for cloudformation:

resource "aws_vpc_endpoint" "cloudformation" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.${var.aws_region}.cloudformation"

  security_group_ids = [
    aws_security_group.elasticbeanstalk_vpc_endpoint.id,
  ]
  
  subnet_ids        = [aws_subnet.private_a.id, aws_subnet.private_b.id]
  
  private_dns_enabled = true

  vpc_endpoint_type = "Interface"
}

• EB gets application zip from S3, yet there is no S3 endpoint. For S3 you can use:

resource "aws_vpc_endpoint" "s3" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.${var.aws_region}.s3"
  
  vpc_endpoint_type = "Gateway"
  
  route_table_ids = [ aws_route_table.private.id ]
}

• there is no vpc endpoint to elasticbeanstalk-healthd. Can use:

resource "aws_vpc_endpoint" "elasticbeanstalk-hc" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.${var.aws_region}.elasticbeanstalk-health"

  security_group_ids = [
    aws_security_group.elasticbeanstalk_vpc_endpoint.id,
  ]
  
  private_dns_enabled = true

  vpc_endpoint_type = "Interface"
}

• if you add new endpoints, you need to update your depends_on in aws_elastic_beanstalk_environment