TCP Load balancer connection refused when reaching from backend service to load balancer

Question

We have a google cloud setup currently that has a backend service (a managed instance group of 3 vm machines) sitting behind a TCP load balancer. In the frontend configuration we have a static external ip address that redirects traffic to port 6443 and a tcp health check on the port 6443 attached to the load balancer as well. Out of the 3 vm instance in the backend service initially only one of the machine has a service running on port 6443. So the load balancer correctly detects one health instance and 2 unhealthy instances in the load balancer details page. In order to bring up a service on the unhealthy vm instances on port 6443 we need to connect to the healthy vm instance via the load balancer ip and the same port. If we connect to the vm instances using the load balancer ip and the port 6443 from outside the load balancer (not from the backend instances) we can see the connection going through successfully to the health vm instance in the backend service. However when trying to connect to the load balancer from one of the unhealth vm instances from the backend service we can see that the connection is re-directed back to the same instance instead of the healthy vm instance causing a connection refused error. Is there a setting in google cloud tcp load balancer to allow connections on the load balancer ip and service port to be always redirected to one of the healthy instances even if the source of the request is from behind the load balancer?

Answer 1

Any LB traffic from a backend will get looped back by the guest OS. The external routing mechanism does not see this traffic and this behavior cannot be changed. This is because the LB IP is part of the local routing table and the OS will always loopback destinations in this table. This is regardless of the backend's health state.

Answer 2

In terms of general network this is not possible.

As a basic example, suppose you have 3 internal private IP addresses, lets call them 192.168.1.2, 192.168.1.3, and 192.168.1.4 for each of your VMs.

The say the are all load balanced using a shared Virtual IP address of 192.168.1.5.

Now your router sits at 192.168.1.1 and it has a public IP of 10.10.10.1 and has been configured to direct incoming external traffic for a given port, say external port 80 for HTTP traffic to the the load balanced IP on some private port, say 6443 for your use case.

Suppose 192.168.1.2, 192.168.1.3 are not healthy and 192.168.1.4 is.

192.168.1.2 sends a request to 10.10.10.1:80.

Your router will now pick up that traffic on 192.168.1.1 and send out a packet to 10.10.10.1:80.

Now the packet comes in on the routers interface at 10.10.10.1:80 and router then routers the packet, per your NAT/Port Forward rules to 192.168.1.5:6443.

The packet goes out on the wire to 192.168.1.5:6443 and since the only healthy host is 192.168.1.4, the packet gets processed by this host.

Answer 3

As I understand, you have configured a TCP load balancer with MIG as the backend.

In TCP load balancer,it routes original connections directly from clients to the healthy backends, without any interruption and responses from the backend VMs go directly to the clients, not back through the load balancer.

It's expected behavior that If the client VM is a backend VM of the load balancer and connections sent to the load balancer's IP address forwarding rule are always answered by the backend VM itself, this happens regardless of whether the backend VM is healthy or not. It happens for all traffic sent to the load balancer's IP address, not just traffic on the protocol and ports specified in the load balancer's forwarding rule.

For troubleshooting purposes, I would suggest you add another test VM within a different VPC network in your project and try to connect to load balancer external IP address from the test VM.