Unable to connect to Elastic Search intermittently

Question

I am trying to connect to elastic search via Jest Client.

Sometimes, the client is not able to connect to the elastic search cluster.

Stack Trace :

org.apache.http.NoHttpResponseException: search-xxx-yyy.ap-southeast-1.es.amazonaws.com:443 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)

The elastic search cluster is in a public domain, so I am not understanding why the client is unable to connect.

Also, the issue happens intermittently, if I retry the request, it connects sometimes.

Any help is appreciated. Thanks

Answer 1

When JestClient initiates the http request, it will call read() on the socket and block. When this read returns -1, this means that the server closed the connection before or during client was waiting for the response.

Why it happens

There's two main causes for NoHttpResponseException:

. The server end of the connection was closed before the client attempts to send a request down it.

. The server end of the connection closes the connection in the middle of a request.

Stale Connection (connection closed before request)

Most often this is a stale connection. When using persistent connections, you may have a connection sit around in the connection pool not being used for a while. If it is idle for longer than the server or load balancer's HTTP keep alive timeout, then the server or load balancer will close the connection due to its idleness. The Jakarta client isn't structured to receive a notification of this happening (it doesn't use NIO), so the connection sits around in a half-closed state. The only way the client can detect this state is by reading from the socket. So when you send a request, the write is successful because the socket is only half closed (writes succeed until you close your end) but then the read indicates the socket was closed. This causes the request to fail.

Connection Closed Mid-Request

The other reason this might occur is the connection was actually closed while the service was working on it. Anything between your client and service may close the connection, including load balancers, proxies, or the HTTP endpoint fronting your service. If your activities are quite long-running or you're transferring a lot of data, the window for something to go wrong is larger and the connection is more likely to be lost in the middle of the request. An example of this happening is a Java server process exiting after an OutOfMemoryException occurs due to trying to return a large amount of data. You can verify whether this is the problem by looking at TCP dumps to see whether the connection is closed while the request is in flight. Also, failures of this type usually occur some time after sending the request, whereas stale connection failures always occur immediately when the request is made.

Diagnosing The Cause

NoHttpResponseException is usually a stale connection (according to problems I've observed and helped people with) When the failure always occurs immediately after submitting the request, stale connection is almost certainly the problem When failures occur some non-trivial amount of time after waiting for the response, then the connection wasn't stale when the request was made and the connection is being closed in the middle of the request TCPDumps can be more conclusive. You can see when the connection is being closed (before or during the request).

What can be done about it

Use a better client

Nonblocking HTTP clients exist that allow the caller to know when a connection is closed without having to try to read from the connection.

Retry failed requests

If your call is safe to retry (e.g. it's idempotent), this is a good option. It also covers all sorts of transient failures besides stale connection failures. NoHttpResponseException isn't necessarily a stale connection and it's possible that the service received the request, so you should take care to retry only when safe.