Cannot connect to RDS in another VPC via VPC peering

Question

I have two VPCs:

• VPC A
  • RDS instance
• VPC B
  • EC2 instance

There are also few subnets:

• VPC A
  • Private A
  • Private B
  • Peer A
• VPC B
  • Private A
  • Private B
  • Peer A

The RDS is in Private A, Private B, Peer A of VPC A.

The EC2 is in Peer A of VPC B.

I want to connect to the RDS instance from the EC2.

I have created a peering:

resource "aws_vpc_peering_connection" "a_to_b" {
  vpc_id      = aws_vpc.a.id
  peer_vpc_id = aws_vpc.b.id
  auto_accept = true

  accepter {
    allow_remote_vpc_dns_resolution = true
  }

  requester {
    allow_remote_vpc_dns_resolution = true
  }
}

resource "aws_vpc_peering_connection_accepter" "a_to_b" {
  vpc_peering_connection_id = aws_vpc_peering_connection.a_to_b.id
  auto_accept               = true
}

I also have route tables for the whole CIDR block like so:

resource "aws_route_table" "a_peer" {
  vpc_id = aws_vpc.a.id
}

resource "aws_route_table_association" "a_peer" {
  route_table_id = aws_route_table.a_peer.id
  subnet_id      = aws_subnet.a_peer.id
}

resource "aws_route" "a_peer_b" {
  route_table_id            = aws_route_table.a_peer.id
  destination_cidr_block    = aws_subnet.b_peer.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.a_to_b.id
}

resource "aws_route_table" "b_peer" {
  vpc_id = aws_vpc.b.id
}

resource "aws_route_table_association" "b_peer" {
  route_table_id = aws_route_table.b_peer.id
  subnet_id      = aws_subnet.b_peer.id
}

resource "aws_route" "b_peer_a" {
  route_table_id            = aws_route_table.b_peer.id
  destination_cidr_block    = aws_subnet.a_peer.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.a_to_b.id
}

I have also created security groups from ingress and egress on the RDS instance to the EC2 security group.

When I SSH into the EC2 I can get the DNS:

$ nslookup rds.xxxxxxxxxxx.eu-west-2.rds.amazonaws.com
Server:     192.16.0.2
Address:    192.16.0.2#53

Non-authoritative answer:
Name:   rds.xxxxxxxxxxx.eu-west-2.rds.amazonaws.com
Address: 10.16.192.135

However, curl cannot connect:

$ curl rds.xxxxxxxxxxx.eu-west-2.rds.amazonaws.com:5432

The expected response is:

$ curl rds.xxxxxxxxxxx.eu-west-2.rds.amazonaws.com:5432
curl: (52) Empty reply from server

The VPC peering is "Active" and the route tables match the Terraform.

How can I get this to connect?

Answer 1

I did some tests on my own, and I'm pretty sure that the issue is caused by your routes, assuming that everything else in your VPC is correct as the VPCs and subnets definitions are not shown .

Specifically, you wrote that "RDS is in Private A, Private B, Peer A of VPC A". This means that RDS master may be in any of these subnets. You have no control over it, as its up to RDS to choose which subnet to use. You can only partially control it by selecting AZs when you create your RDS. Subsequently, your peering route tables should cover all these three subnets. The easiest way to achieve this is by using VPC CIDR range:

# Route from instance in VPC B to any subnet in VPC A which
# hosts your RDS in all its subnets
resource "aws_route" "b_peer_a" {
  route_table_id            = aws_route_table.b_peer.id
  destination_cidr_block    = aws_vpc.a.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.a_to_b.id
}

Then you also need to have a route table in VPC A associated with your peering connections for all its subnets:

resource "aws_route_table" "a_peer" {
  vpc_id = aws_vpc.a.id
}

resource "aws_route_table_association" "a_peer" {
  route_table_id = aws_route_table.a_peer.id
  subnet_id      = aws_subnet.a_peer.id
}

resource "aws_route_table_association" "a_private1" {
  route_table_id = aws_route_table.a_peer.id
  subnet_id      = aws_subnet.a_private1.id
}

resource "aws_route_table_association" "a_private2" {
  route_table_id = aws_route_table.a_peer.id
  subnet_id      = aws_subnet.a_private2.id
}

resource "aws_route" "a_peer_b" {
  route_table_id            = aws_route_table.a_peer.id
  destination_cidr_block    = aws_subnet.b_peer.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.a_to_b.id
}