Neo4j OGM Bolt Java driver throws "connection reset by peer" exception

Question

Like mentioned here we have almost the same issues with a java project (playframework) with neo4j-ogm and the ogm-bolt-driver (3.2.27). Connection resets sometimes, throws exception and reconnects with the next request (mostly). The nodes will be queried from database via repositories and neo4j-ogm session (will be opened for small pieces of work).

Any advice will be appreciated.

play.api.UnexpectedException: Unexpected exception[ConnectionException: Connection to the database failed]
    at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:358)
    at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:264)
    at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:430)
    at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:422)
    at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:454)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: org.neo4j.ogm.exception.ConnectionException: Connection to the database failed
    at org.neo4j.ogm.drivers.bolt.driver.BoltDriverExceptionTranslator.translateExceptionIfPossible(BoltDriverExceptionTranslator.java:38)
    at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:601)
    at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:558)
    at org.neo4j.ogm.session.delegates.LoadOneDelegate.load(LoadOneDelegate.java:87)
    at org.neo4j.ogm.session.delegates.LoadOneDelegate.load(LoadOneDelegate.java:53)
    at org.neo4j.ogm.session.Neo4jSession.load(Neo4jSession.java:178)
    at modules.shared.repositories.CrudRepository.find(CrudRepository.java:50)
    at config.secured.Secured.getUserFromSession(Secured.java:88)
    at config.secured.Secured.getUser(Secured.java:56)
    at config.secured.Secured.getUser(Secured.java:39)
    at config.secured.DataWizardSecurity$AuthenticatedAction.call(DataWizardSecurity.java:63)
    at config.filter.SafeFormFactoryToRequest.call(SafeFormFactoryToRequest.java:24)
    at config.filter.Messaged.call(Messaged.java:29)
    at play.core.j.JavaAction.$anonfun$apply$8(JavaAction.scala:175)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:672)
    at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:431)
    at play.core.j.HttpExecutionContext.$anonfun$execute$1(HttpExecutionContext.scala:64)
    at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:70)
    at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:59)
    at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:393)
    at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:302)
    at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:276)
    at scala.concurrent.impl.Promise$DefaultPromise.map(Promise.scala:146)
    at scala.concurrent.Future$.apply(Future.scala:672)
    at play.core.j.JavaAction.apply(JavaAction.scala:176)
    at play.api.mvc.Action.$anonfun$apply$4(Action.scala:82)
    at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:168)
    at scala.util.Try$.apply(Try.scala:210)
    at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:168)
    at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
    at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
    at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
    at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
    at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
    at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:199)
    at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:417)
    at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
    at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51)
    at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:448)
    ... 12 common frames omitted
Caused by: org.neo4j.driver.exceptions.ServiceUnavailableException: Connection to the database failed
    at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:143)
    at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:98)
    at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:92)
    at org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction.newOrExistingNativeTransaction(BoltTransaction.java:62)
    at org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction.<init>(BoltTransaction.java:50)
    at org.neo4j.ogm.drivers.bolt.driver.BoltDriver.lambda$null$0(BoltDriver.java:128)
    at org.neo4j.ogm.session.transaction.DefaultTransactionManager.openTransaction(DefaultTransactionManager.java:75)
    at org.neo4j.ogm.session.Neo4jSession.beginTransaction(Neo4jSession.java:530)
    at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:580)
    ... 49 common frames omitted
    Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause: null
        at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.transformError(ChannelErrorHandler.java:127)
        at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.fail(ChannelErrorHandler.java:113)
        at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.exceptionCaught(ChannelErrorHandler.java:99)
        at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
        at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
        at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
        at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
        at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
        at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
        at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
        at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
        at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
        at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
        at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
        at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.net.SocketException: Connection reset
    at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:345)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:376)
    at org.neo4j.driver.internal.shaded.io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
    at org.neo4j.driver.internal.shaded.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
    at org.neo4j.driver.internal.shaded.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
    at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
    at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:832)

Answer 1

As I mention in my answer here, this exception comes from Neo4j Java Driver and there may be several reasons for this to happen.

For instance, this might happen when driver acquires a connection from internal connection pool that has been idle for too long. Some cloud providers limit connection idle time for load balancers. Driver may fail as soon as it tries using such connection.

You can enable connection liveness check to assist with this. It makes sure that a short network exchange happens for every connection that has been idle over the configured timeout. If exchange fails, another connection is picked or established before it is provided for further driver usage. While this involves an extra network exchange, it is likely to prevent driver failures that might lead to higher-level retry management with potentially exponential delay. The timeout can be set to zero if you wish to test every connection.

Sample configuration:

var config = Config.builder()
                   .withConnectionLivenessCheckTimeout( 60000, TimeUnit.MILLISECONDS )
                   .build();

var driver = GraphDatabase.driver( URL, AuthTokens, config );