CODEWITHSUNDEEP
c#httpclientbad-gatewayservicepointmanager
RickySpanish
RickySpanish

Reputation: 511

HttpClient stops re-using connections after load balancer change

I have a client that connects to a cloud server setup - a HAProxy instance (one of several, determined by a load balancer), which will then forward the requests onto the correct backend server.

From the client perspective, this is just one constant URI, but an issue occurs when the load balancer switches the HAProxy instance that handles the request. After this happens, every subsequent request HttpClient sends is from a different local port and is treated by the backend server as a new connection.

A persistent connection is critical to the application, so this majorly breaks it. Strangely, the issue resolves itself as soon as the client application is restarted. This leads me to believe the problem lies with the client, and not the cloud setup or backend server.

Is there something I'm missing here? I figured it might be a problem with ServicePointManager creating a ServicePoint when the connection is going through one HAProxy instance, and since this object is maintained for the life of the application, perhaps there is some issue with it being used through the second HAProxy instance. When the application is restarted, the ServicePoint instance is lost and re-created.

Update: I noticed that the ServicePoint object has HTTP Protocol 1.1 at first, but once the issue occurs it has changed to 1.0. HTTP 1.1 has persistent connections by default, while HTTP 1.0 has temporary connections by default.

If I change the ServicePoint object protocol to 1.0 when everything is working, I will get a 502 Bad Gateway exception (although it seems to recover and change back to 1.1 for the next request).

I'm guessing something in the cloud stack is causing the protocol to change to 1.0 after load balancing.

Upvotes: 0

Views: 976

Answers (1)

RickySpanish
RickySpanish

Reputation: 511

It seems like something in the cloud stack was setting the response HTTP version to 1.0 when the load balancer switch caused the initial error. The client's ServicePoint object for my endpoint then got updated so that ServicePoint.HttpBehaviour and ServicePoint.ProtocolVersion were both 1.0.

HTTP 1.1 has persistent connections by default, while 1.0 does not, so connections were not kept alive.

Changing ServicePoint.HttpBehaviour back to 1.1 fixes the issue, although I'm sure there's a proper configuration fix that can be done on the cloud server.

One issue is that HttpBehaviour and ProtocolVersion are private properties (I was changing via visual studio debugger's quick watch). They should be changeable via reflection, although then I'm dependent on it not being changed in future .NET versions.

Upvotes: 1

Related Questions