ssh to private ec2 instance behind internet-facing NLB

Question

Here's my setup:

1. I have 3 instances in us-east-1a
   • instanceA is in public subnet with publicIP
   • instanceB is in private subnet without publicIP
   • instanceC is in public subnet with publicIP
2. I put instanceC behind NLB. The NLB is on public subnets us-east-1a and us-east-1b
3. The security group for instanceC is:
   • Allow SSH from instanceA's public IP
   • Allow SSH from instanceB's private IP

I can directly ssh to instanceC from instanceA and B without any issue. But whenever I ssh to the NLB's DNS, i can only connect from instanceA. Whenever I ssh to NLB from instanceB, there will just be a connection timed out.

I checked the network interface of the NLB, and I see that there are a pair of public and private IP per NLB's availability zone.

From instanceA, I can ssh to NLB's public IP addresses, but I can not ssh to NLB's privateIPs. From instanceB, I can ssh to NLB's private IP addresses, but I can not ssh to NLB's publicIPs

Now, i tried to ssh -vvv the NLB, and I can see that I am trying to connect to NLB's public IP. This might be the reason why I can not connect to NLB from instanceB because as I said above, instanceB can only connect to NLB's privateIPs.

I tried to modify the sg of InstanceC and allow ssh from 0.0.0.0/0 and I can successfully access the NLB from instanceB. But this is not what I want. I don't want to allow everyone to access instanceC via NLB.

I want instanceB to be able to ssh to instanceC via NLB too just like instanceA. How can I accomplish this? What should I need to modify in sg, nlb, az, etc?

Thanks!

Answer 1

Thanks Nael and John!

I forgot to mention that each instances belong to different VPC. InstanceA and NLB belongs to same VPC.

I was able to solve my problem by granting ssh access to natgateway publicIP of instanceB's VPC. Doing that allows me to ssh NLB's public IP from instanceB.

Answer 2

To just address one aspect from your question...

You should avoid putting IP addresses of Amazon EC2 instances into a security group. If you wish to allow two instances to communicate (eg Instance A talking to Instance C), you should configure:

• A Security Group on Instance A (SG-A), allowing incoming traffic (presumably) from the Internet
• A Security Group on Instance C (SG-C), that allows inbound traffic from SG-A

That is, SG-C should specifically reference SG-A. This reference will be automatically converted into the Security Group ID (eg sg-xxx). This means that any instance associated with SG-A will be permitted inbound connections on the given port to any instance associated with SG-C.

The benefit of this approach is that instances can be replaced, or change IP address, without impacting the security groups. You can also add additional instances into these security groups and they will be automatically granted the same permissions.

Bottom line: Avoid referencing specific private IP addresses within a security group.

Answer 3

I think by adding the security group of instance B to the security group of instance C you could ssh to instance C via NLB's with instance B:

So add an inbound rule to sg of instance C like shown in the picture and specify sg of instance B