CODEWITHSUNDEEP
socketsamazon-web-servicestcpamazon-elb
Magnus
Magnus

Reputation: 8310

AWS TCP ELB refuse connection when there is no available back-end server

We have a TCP application that receives connections in a protocol that we did not design and don’t control. This protocol will assume that if it can establish a TCP connection, then it can send a message and that message is acknowledged.

This works ok if connecting directly to a machine, if the machine or application is down, the tcp connection will be refused or dropped and the client will attempt to redeliver the message.

When we use AWS elastic load balancer, ELB will establish a TCP connection with the client, regardless of whether there is an available back-end server to fulfil the request. As a result if our application or server crashes then we lose messages.

ELB will close the TCP connection shortly thereafter, but its not good enough.

Is there a way to make ELB, only establish a connection if it can reach the back-end server? What options do we have (within the AWS ecosystem), of balancing a TCP based service, while still refusing connections if they cannot be served.

Upvotes: 0

Views: 2859

Answers (2)

Mircea
Mircea

Reputation: 10566

I don't think that's achievable through ELB. By design a load balancer will manage 2 sets of connections (frontend - LB and LB - backend). The load balancer will attempt to minimize the time it takes to serve the traffic it receives. This means that the FE-LB connection will be established as the LB looks for a Backend connection to use / reuse. The case in which all of the Backend hosts are dead is such an edge case that you end up with the behavior you are seeing. Normally it's not a big deal as the requested will just get disconnected once the LB figures out that it cannot server the traffic.

Back to your protocol: to me it seem really weird that you would interpret the ability to establish a connection as equal to message delivery. It sounds like you're using TCP but not waiting for the confirmations that the message were actually received at the destination. To me that seems wrong and will get you in trouble eventually with or without a load balancer.

And not to sound too pessimistic (I do understand we are not living in an ideal world) what I would do in this specific scenario, if you can deploy additional software on the client, would be to use a tcp proxy on the client that would get disabled automatically whenever the load balancer is unhealthy/unable to serve traffic. Instruct the client to connect to this proxy. Far from ideal but it should do the trick.

Upvotes: 2

Andrei G
Andrei G

Reputation: 76

You could create a health check from your ELB to verify if the backend EC2 instances respond on the TCP port. See ELB Health Checks

Then, you monitor the health status of the EC2 instances sent by the ELB to CloudWatch.

Once you determine that none of the EC2 instances are responding on the TCP port, you can remove the TCP listener from the ELB. See Delete ELB Listeners

Hopefully, at that point the ELB stops accepting TCP connections.

Note, I have not tested this solution.

Upvotes: 0

Related Questions