Communications link failure: Connection reset with C3P0

Question

I know this has been asked many a times but I am stuck at it.

I verified following things:

IP address or hostname in JDBC URL is correct.

Hostname in JDBC URL is also recognized by local DNS server.

Port number is correct in JDBC URL.

DB server is up.

Since same DB call works sometimes, I believe it to relate with "Out of Connections". We had only 50-60 connections to DB at the time of issue.

Still, I believe that C3PO connections might have exhausted.

We have following C3P0 configurations

<props>
<prop key="hibernate.c3p0.timeout">
44000
<!--  seconds  -->
</prop>
<prop key="hibernate.c3p0.min_size">5</prop>
<prop key="hibernate.c3p0.max_size">5</prop>
<prop key="hibernate.c3p0.initial_pool_size">5</prop>
<prop key="hibernate.c3p0.idle_test_period">
60
<!--  seconds  -->
</prop>
<prop key="hibernate.c3p0.acquire_increment">2</prop>
<prop key="hibernate.c3p0.max_statements">50</prop>
<prop key="hibernate.c3p0.preferredTestQuery">SELECT 1 FROM DUAL</prop>
<prop key="hibernate.show_sql">true</prop>
</props>

Most of the entries are not defined in here and default is used. Let me know if this might be the issue and reason would be very helpful.

Maximum number C3P0 connections by default are 15 but we almost cross this limit all the time and don't face this issue (again it might be bcoz of acquireIncrement property of C3P0 to support connections in this scenario).

Stacktrace:

org.springframework.dao.DataAccessResourceFailureException: could not execute query; nested exception is org.hibernate.exception.JDBCConnectionException: could not execute query
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:625)
at org.springframework.orm.hibernate3.AbstractSessionFactoryBean.convertHibernateAccessException(AbstractSessionFactoryBean.java:303)
at org.springframework.orm.hibernate3.AbstractSessionFactoryBean.translateExceptionIfPossible(AbstractSessionFactoryBean.java:282)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:58)
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:213)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:163)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy34.findByName(Unknown Source)
at a.b.c.d.e.f(A.java:941)
at a.b.c.d.e.f$$FastClassByCGLIB$$a171eb15.invoke(<generated>)
at org.springframework.internal.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at com.amazon.mobiusdatabase.activity.IMBatchActivity$$EnhancerByCGLIB$$42d90986.getLatestBatchByStatus(<generated>)
at com.amazon.coral.reflect.$2198.invoke(Unknown Source)
at com.amazon.coral.service.DefaultActivityInvoker.invoke(DefaultActivityInvoker.java:40)
at com.amazon.coral.service.ContinuationActivityInvokerFactory$1.invoke(ContinuationActivityInvokerFactory.java:92)
at com.amazon.coral.service.ActivityHandler.invoke(ActivityHandler.java:269)
at com.amazon.coral.service.ActivityHandler.before(ActivityHandler.java:157)
at com.amazon.coral.service.TransmutingContinuationHandler.before(TransmutingContinuationHandler.java:51)
at com.amazon.coral.service.Chain.componenteBefore(Chain.java:98)
at com.amazon.coral.service.Chain.before(Chain.java:76)
at com.amazon.coral.service.helper.ChainHelper.before(ChainHelper.java:103)
at com.amazon.coral.service.AbstractOrchestrator.doWork(AbstractOrchestrator.java:60)
at com.amazon.coral.service.PassiveOrchestrator.enqueue(PassiveOrchestrator.java:68)
at com.amazon.coral.service.GracefulShutdownOrchestrator.enqueue(GracefulShutdownOrchestrator.java:79)
at com.amazon.coral.service.helper.OrchestratorHelper.enqueue(OrchestratorHelper.java:44)
at com.amazon.coral.service.batch.BatchProtocolOrchestrator.enqueue(BatchProtocolOrchestrator.java:97)
at com.amazon.coral.service.bobcat.BobcatEndpoint.invoke(BobcatEndpoint.java:194)
at org.apache.catalina.valves.SSLValve.invoke(SSLValve.java:113)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:894)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:719)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2101)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 1 ms ago.
at sun.reflect.GeneratedConstructorAccessor143.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2985)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2871)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3414)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
... 49 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2428)

We are getting this while firing a select query on DB.

Answer 1

Maybe the connections aren't correctly closed, so you can try to decrease drastically the timeout to see if you have the same problem.

 <prop key="hibernate.c3p0.timeout">3000</prop>

At the same time you could use a JMX profiler like VisualVM to monitor the C3P0 connections, and see directly on the database server if there are "lost" connections.