CODEWITHSUNDEEP
javaspringelasticsearchspring-data-elasticsearch
Fedor
Fedor

Reputation: 669

ES Rest High Level Client throws SocketTimeoutException after being idle for sometime

RestHighLevelClient is used to connect to ES 6.4(hosted on AWS) in a spring-boot app. When the app goes idle for some time, and request arrives, then the RestHighLevelClient throws SocketTimeoutException:

[Request processing failed; nested exception is org.springframework.data.elasticsearch.ElasticsearchException: Error while bulk for request: org.elasticsearch.action.bulk.BulkRequest@21511b6c] w
java.net.SocketTimeoutException: 5,000 milliseconds timeout on connection http-outgoing-38 [ACTIVE]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

RestHighLevelClient is created using:

    @Bean
    RestHighLevelClient client() {

        ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo(elasticsearchHostAndPort)
                .build();

        return RestClients.create(clientConfiguration).rest();
    }

spring-data-elasticsearch version 3.2.0.M2 is used.

Any hints/workarounds?

Upvotes: 8

Views: 14589

Answers (5)

user12428264
user12428264

Reputation: 11

 @Bean
override fun elasticsearchClient(): RestHighLevelClient {
    val clientConfiguration = ClientConfiguration
        .builder()
        .connectedTo("localhost:9200")
        .withSocketTimeout(enter time in millisecond)
        .build()`enter code here`
    return RestClients.create(clientConfiguration).rest()
}

just use withSocketTimeout(30000) mthode like shown above

Upvotes: 0

orid
orid

Reputation: 329

I have also tried setting the connection/socket timeout to 0, as suggested here and in other places. It didn't help eventually.

There is another solution/workaround, suggested by spring-data-elasticsearch in https://jira.spring.io/browse/DATAES-789. It simply perform an internal retry in case of such exception. It does not really solve the issue, but your client will not get and error. Instead, the first request after idle time will take additional 5 seconds (or whatever timeout your configured).

If you use Springboot-data-elasticsearch version 4+ (staring with Springboot 2.3.0), then you can apply the solution.

I can confirm the following solution is working (with the limitations I mentioned above):


@Configuration
public class ElasticSearchRestClientConfig extends AbstractElasticsearchConfiguration {

    @Autowired
    private RestHighLevelClient restHighLevelClient;

    @Override
    public RestHighLevelClient elasticsearchClient() {
        return restHighLevelClient;
    }

    @Bean
    @Override
    public ElasticsearchCustomConversions elasticsearchCustomConversions() {
        return new ElasticsearchCustomConversions();
    }

    @Override
    public ElasticsearchOperations elasticsearchOperations(ElasticsearchConverter elasticsearchConverter) {
        return new ElasticsearchRestTemplate(elasticsearchClient(), elasticsearchConverter) {
            @Override
            public <T> T execute(ClientCallback<T> callback) {
                int retryCount = 0;
                T t = null;
                while (retryCount <= RestClientBuilder.DEFAULT_MAX_CONN_PER_ROUTE && t == null) {
                    try {
                        t = super.execute(callback);
                    } catch (DataAccessResourceFailureException e) {
                        // retry
                        if (e.getCause() != null && (e.getCause().getCause() instanceof SocketTimeoutException) &&
                                (retryCount < RestClientBuilder.DEFAULT_MAX_CONN_PER_ROUTE)) {
                            retryCount++;
                            log.warn("Elasticsearch client - performing retry {} after caught DataAccessResourceFailureException: {}", retryCount, e.getMessage());
                        }
                        else {
                            throw e;
                        }
                    }
                }
                return t;
            }
        };
    }

Upvotes: 7

Doston
Doston

Reputation: 637

Here is how I solved this issue:

@Configuration
@EnableElasticsearchRepositories(basePackages = "com.hamdos.repositories.es")
@ComponentScan(basePackages = { "com.hamdos.services.es" })
@Order(2)
public class Config {

    @Bean
    RestHighLevelClient client() {

        RestClientBuilder builder = RestClient.builder(
                new HttpHost("localhost", 9289))
                .setRequestConfigCallback(
                        new RestClientBuilder.RequestConfigCallback() {
                            @Override
                            public RequestConfig.Builder customizeRequestConfig(
                                    RequestConfig.Builder requestConfigBuilder) {
                                return requestConfigBuilder
                                        .setConnectTimeout(5000)
                                        .setSocketTimeout(60000);
                            }
                        });

        return new RestHighLevelClient(builder);
    }

    @Bean
    public ElasticsearchOperations elasticsearchTemplate() {
        return new ElasticsearchRestTemplate(client());
    }
}

Info about system:

  • spring-data-elasticsearch: 4.0.0.RELEASE
  • spring-boot: 2.3.0.RELEASE
  • Elasticsearch: 7.7.0

Upvotes: 1

Eljah
Eljah

Reputation: 5154

it should be also configurable thru properties:

spring.elasticsearch.jest.connection-timeout=3000 # Connection timeout in milliseconds.
spring.elasticsearch.jest.multi-threaded=true # Enable connection requests from multiple execution threads.

Upvotes: 0

Philipp Wirth
Philipp Wirth

Reputation: 920

In RestClientBuilder.createHttpClient() the defaults for socket timeout and connection timeout are set to 30 and 10 seconds.

You can override this defaults by implementing RestClientBuilder.RequestConfigCallback and calling setRequestConfigCallback(...) on your RestHighLevelClient

We did something like

@Override
public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder builder) {
    return builder.setSocketTimeout(socketTimeout); // try to prevent SocketTimeoutException
}

Upvotes: 3

Related Questions