ElasticSearch RestHighLevelClient throws java.io.IOException: Connection reset by peer

Question

Summary

When my Spring Boot app has been idle for a while, calling the search method of an ElasticsearchRepository (which uses the RestHighLevelClient for the underlying ES connection) results in a java.io.IOException: Connection reset by peer being thrown.

---

Detailed problem

I have a Spring Boot app (version 2.2.4.RELEASE) and an ElasticSearch (version 6.8.6). For communication with the ES instance I use Spring Data Elasticsearch (version 3.2.5.RELEASE), to be specific I use an ElasticsearchRepository.

My ES configuration class looks as follows:

@Configuration
@EnableElasticsearchRepositories
public class ElasticsearchConfiguration {

    private final RestHighLevelClient restHighLevelClient;

    @Autowired
    public ElasticsearchConfiguration(RestHighLevelClient restHighLevelClient) {
        this.restHighLevelClient = restHighLevelClient;
    }

    @Bean
    public ElasticsearchRestTemplate elasticsearchTemplate() {
        return new ElasticsearchRestTemplate(restHighLevelClient);
    }

}

I have got an REST endpoint which triggers a lookup in the ES by calling the below method:

@Service
@Transactional(readOnly = true)
public class SportsFacilityViewFilterService {

    private final SportsFacilityViewRepository sportsFacilityViewRepository;

    @Autowired
    public SportsFacilityViewFilterService(SportsFacilityViewRepository sportsFacilityViewRepository) {
        this.sportsFacilityViewRepository = sportsFacilityViewRepository;
    }

    /**
     * Retrieves all {@link SportsFacilityView}s stored in the search index that match the given filter parameters.
     *
     * @param activity filter parameter.
     * @param city     filter parameter.
     * @param date     filter parameter.
     * @param capacity filter parameter.
     * @return {@code List<SportsFacilityView>} the filtered sports facilities in a display optimized format.
     */
    @Retryable(include = IOException.class, maxAttempts = 1, backoff = @Backoff(delay = 15))
    @Transactional(readOnly = true)
    public List<SportsFacilityView> filterSportsFacilities(final String activity, final Integer capacity,
                                                           final String city, final LocalDate date) {
        // Use filter query to retrieve sports facility views
        final NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
            .withFilter(buildFilterQuery(activity, capacity, city))
            .withPageable(PageRequest.of(0, 50))
            .withSort(new FieldSortBuilder("vendorName").order(SortOrder.ASC))
            .build();
        final List<SportsFacilityView> sportsFacilities = Lists.newArrayList(sportsFacilityViewRepository.search(searchQuery));

        // Post-process filter results
        removeClosedSportsFacilities(sportsFacilities, date);
        updateMinimumPrice(sportsFacilities, date);

        // Return
        return sportsFacilities;
    }

}

I defined the SportsFacilityViewRepository as follows:

@Repository
public interface SportsFacilityViewRepository extends ElasticsearchRepository<SportsFacilityView, Long> {

}

This code works perfectly fine, but when the system has been idle for a while, the following exception is thrown when invoking the above illustrated method:

org.springframework.data.elasticsearch.ElasticsearchException: Error for search request with scroll: SearchRequest{searchType=DFS_QUERY_THEN_FETCH, indices=[sportsfacility], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[sportsfacilityview], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"from":0,"size":50,"post_filter":{"bool":{"filter":[{"term":{"address.city":{"value":"Köln","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"sort":[{"vendorName":{"order":"asc"}}]}}
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doSearch(ElasticsearchRestTemplate.java:1153) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.queryForPage(ElasticsearchRestTemplate.java:381) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.queryForPage(ElasticsearchRestTemplate.java:376) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.queryForPage(ElasticsearchRestTemplate.java:147) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.search(AbstractElasticsearchRepository.java:259) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at sun.reflect.GeneratedMethodAccessor260.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:371) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:204) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:657) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:621) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:605) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at com.sun.proxy.$Proxy239.search(Unknown Source) ~[na:na]
        at sun.reflect.GeneratedMethodAccessor259.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at com.sun.proxy.$Proxy239.search(Unknown Source) ~[na:na]
        at com.myapp.service.SportsFacilityViewFilterService.filterSportsFacilities(SportsFacilityViewFilterService.java:74) ~[app/:na]
        at com.myapp.service.SportsFacilityViewFilterService$$FastClassBySpringCGLIB$$d9ad847c.invoke(<generated>) ~[app/:na]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$1117.00000000674D8570.proceedWithInvocation(Unknown Source) ~[na:na]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366) ~[spring-tx-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:99) ~[spring-tx-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at com.myapp.service.SportsFacilityViewFilterService$$EnhancerBySpringCGLIB$$492633ab.filterSportsFacilities(<generated>) ~[app/:na]
        at com.myapp.web.controller.publicapi.SportsFacilityFilterController.filterSportsFacilities(SportsFacilityFilterController.java:42) ~[app/:na]
        at com.myapp.web.controller.publicapi.SportsFacilityFilterController$$FastClassBySpringCGLIB$$13529b2a.invoke(<generated>) ~[app/:na]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:120) ~[spring-context-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at com.myapp.web.controller.publicapi.SportsFacilityFilterController$$EnhancerBySpringCGLIB$$221bc350.filterSportsFacilities(<generated>) ~[app/:na]
        at sun.reflect.GeneratedMethodAccessor258.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209) [spring-security-web-5.2.1.RELEASE.jar:5.2.1.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.2.1.RELEASE.jar:5.2.1.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_222]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_222]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at java.lang.Thread.run(Thread.java:819) [na:1.8.0_222]
Caused by: java.io.IOException: Connection reset by peer
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092) ~[elasticsearch-rest-high-level-client-6.8.6.jar:6.8.6]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doSearch(ElasticsearchRestTemplate.java:1151) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        ... 108 common frames omitted
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_222]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_222]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_222]
        at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_222]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_222]
        at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:231) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
        ... 1 common frames omitted

I tried forcing a retry by annotating the method with @Retryable, but unfortunately this appears to be ignored, despite being picked up in other places of the program.

Can somebody please explain why I get this exception and how I can prevent it from occurring?

Update

Removing the @Repository annotation from SportsFacilityViewRepository made it possible to try/catch the code, respectively automatically retry by means of the @Retryable annotation. Obviously, the Spring Exception Translation mechanism applied due to the @Repository annotation being present prevented a try/catch block and Spring Retry to catch the exception that was shown on the console.

Answer 1

Maybe you can config the client like this:

    @Override
    @Bean
    public RestHighLevelClient elasticsearchClient() {

        // Connect to elastic search in https://app.bonsai.io/
        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo(hostAndPort)
                .withBasicAuth(userName, pwd)
                // config the connect
                .withConnectTimeout(5000)
                .withSocketTimeout(60000)
                .build();

        return RestClients.create(clientConfiguration).rest();
    }

Answer 2

Here's some configuration:

 @Configuration
 public class ElasticSearchConfig extendsAbstractElasticsearchConfiguration {
    @Value("${spring.elasticsearch.rest.uris}")
    private String uris;

    @Value("${spring.elasticsearch.rest.username:#{\"\"}}")
    private String username;

    @Value("${spring.elasticsearch.rest.password:#{\"\"}}")
    private String password;

    @Override
    public RestHighLevelClient elasticsearchClient() {

        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo(uris)
                .withBasicAuth(username, password)
                .withHttpClientConfigurer(clientBuilder -> {
                    clientBuilder.setKeepAliveStrategy((httpResponse, httpContext) -> 1000 * 60);
                    return clientBuilder;
                })
                .build();
        return RestClients.create(clientConfiguration).rest();

    }

}

Answer 3

I finally found an easily viable workaround. I let Spring execute a scheduled task which executes the count method of the relevant ElasticsearchRepositories. By that means, the connection is kept alive so that no Exceptions can occur due to the system being idle for too long.

The code looks as follows:

@Component
@Slf4j
public class ElasticsearchConnectionTester {

    private final SportsFacilityViewRepository sportsFacilityViewRepository;

    @Autowired
    public ElasticsearchConnectionTester(SportsFacilityViewRepository sportsFacilityViewRepository) {
        this.sportsFacilityViewRepository = sportsFacilityViewRepository;
    }

    @Scheduled(fixedRate = 1800000, initialDelay = 1800000)
    public void keepConnectionAlive() {
        log.debug("Trying to ping Elasticsearch");
        try {
            final long noOfSportsFacilities = sportsFacilityViewRepository.count();
            log.debug("Ping succeeded for SportsFacilityViewRepository, it contains {} entities", noOfSportsFacilities);
        } catch (Exception e) {
            log.debug("Ping failed for SportsFacilityViewRepository");
        }
    }
}

The logging is optional, but it provides the necessary insight whether the chosen rate for the scheduled job is sufficient.