Shared C3P0 JNDI DataSource gets closed during Jetty's servlet undeployment, no longer accessible to other servlets

Question

Several servlets run inside my Jetty container. All these servlets use one DataSource exposed via JNDI. This DataSource is a C3P0 ComboPooledDataSource.

At the moment I undeploy any of those servlets, somehow the ComboPooledDataSource gets "closed". From this moment on, both the already-deployed servlets and any additionally deployed servlets can no longer access the DataSource. Therefore, all servlets that need that DataSource stop working at the next access to said DataSource.

Here is a commented stacktrace:

## Undeploying a servlet named "c.war" by issuing "rm -f ${jetty.base}/webapps/c.war"
## Thread "Scanner-0" recognizes that something has changed in the webapps directory,
## therefore "Scanner-0" shuts down components inside the c.war servlet:

2013-12-23 17:19:11,977 container [Scanner-0] INFO  c - Destroying Spring FrameworkServlet 'appServlet'
2013-12-23 17:19:11,977 container [Scanner-0] INFO  o.s.w.c.s.XmlWebApplicationContext - Closing WebApplicationContext for namespace 'appServlet-servlet': startup date [Mon Dec 23 17:18:01 CET 2013]; parent: Root WebApplicationContext
2013-12-23 17:19:11,977 container [Scanner-0] INFO  o.s.b.f.s.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@6def78d2: defining beans []; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@477ed07f
2013-12-23 17:19:11,980 container [Scanner-0] INFO  c - Closing Spring root WebApplicationContext
2013-12-23 17:19:11,980 container [Scanner-0] INFO  o.s.w.c.s.AnnotationConfigWebApplicationContext - Closing Root WebApplicationContext: startup date [Mon Dec 23 17:17:37 CET 2013]; root of context hierarchy
2013-12-23 17:19:11,998 container [Scanner-0] INFO  o.s.c.s.DefaultLifecycleProcessor - Stopping beans in phase 2147483647
2013-12-23 17:19:12,003 container [Scanner-0] INFO  org.quartz.core.QuartzScheduler - Scheduler my_scheduler_$_NON_CLUSTERED paused.
2013-12-23 17:19:12,006 container [Scanner-0] INFO  o.s.b.f.s.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@477ed07f: defining beans [long, list, of, my, spring, beans, shortened, for, brevity]; root of factory hierarchy
2013-12-23 17:19:12,007 container [Scanner-0] INFO  org.apache.tiles.access.TilesAccess - Removing TilesContext for context: org.springframework.web.servlet.view.tiles2.SpringTilesApplicationContextFactory$SpringWildcardServletTilesApplicationContext
2013-12-23 17:19:12,014 container [Scanner-0] INFO  o.s.s.quartz.SchedulerFactoryBean - Shutting down Quartz Scheduler
2013-12-23 17:19:12,014 container [Scanner-0] INFO  org.quartz.core.QuartzScheduler - Scheduler my_scheduler_$_NON_CLUSTERED shutting down.
2013-12-23 17:19:12,014 container [Scanner-0] INFO  org.quartz.core.QuartzScheduler - Scheduler my_scheduler_$_NON_CLUSTERED paused.
2013-12-23 17:19:12,056 container [Scanner-0] INFO  org.quartz.core.QuartzScheduler - Scheduler my_scheduler_$_NON_CLUSTERED shutdown complete.
2013-12-23 17:19:12,155 container [Scanner-0] INFO  o.e.j.server.handler.ContextHandler - Stopped o.e.j.w.WebAppContext@676e4e1c{/c,file:/tmp/jetty-0.0.0.0-8080-c.war-_c-any-8486076679973394405.dir/webapp/,UNAVAILABLE}{/c.war}

## Thread "Scanner-0" has finished undeploying c.war

## A few seconds later, thread "my_scheduler_QuartzSchedulerThread" from unrelated 
## servlet "b.war" tries to do stuff with the JNDI-obtained DataSource, but fails:

2013-12-23 17:19:19,688 container [my_scheduler_QuartzSchedulerThread] ERROR org.quartz.core.ErrorLogger - An error occurred while scanning for the next triggers to fire.
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'springNonTxDataSource.my_scheduler': java.sql.SQLException: com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, dataSourceName -> ai8gp08z8xy71x5vsabw|11f7562b, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, extensions -> {}, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> ai8gp08z8xy71x5vsabw|11f7562b, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://localhost:3306/mydb?useUnicode=true&characterEncoding=UTF-8, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 28000, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 40, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 20, numHelperThreads -> 3, preferredTestQuery -> null, privilegeSpawnedThreads -> false, properties -> {user=******, password=******}, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {}, usesTraditionalReflectiveProxies -> false ] has been closed() -- you can no longer use it.
    at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168) ~[quartz-2.2.1.jar:na]
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3784) ~[quartz-2.2.1.jar:na]
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756) ~[quartz-2.2.1.jar:na]
    at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272) ~[quartz-2.2.1.jar:na]
Caused by: java.sql.SQLException: com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, dataSourceName -> ai8gp08z8xy71x5vsabw|11f7562b, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, extensions -> {}, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> ai8gp08z8xy71x5vsabw|11f7562b, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://localhost:3306/mydb?useUnicode=true&characterEncoding=UTF-8, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 28000, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 40, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 20, numHelperThreads -> 3, preferredTestQuery -> null, privilegeSpawnedThreads -> false, properties -> {user=******, password=******}, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {}, usesTraditionalReflectiveProxies -> false ] has been closed() -- you can no longer use it.
    at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.assertCpds(AbstractPoolBackedDataSource.java:507) ~[c3p0-0.9.5-pre6.jar:0.9.5-pre6]
    at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getPoolManager(AbstractPoolBackedDataSource.java:519) ~[c3p0-0.9.5-pre6.jar:0.9.5-pre6]
    at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140) ~[c3p0-0.9.5-pre6.jar:0.9.5-pre6]
    at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:129) ~[spring-context-support-3.2.4.RELEASE.jar:3.2.4.RELEASE]
    at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108) ~[quartz-2.2.1.jar:na]
    at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165) ~[quartz-2.2.1.jar:na]
    ... 3 common frames omitted

So the relevant message here is:

An error occurred while scanning for the next triggers to fire.
org.quartz.JobPersistenceException:
Failed to obtain DB connection from data source 'springNonTxDataSource.my_scheduler':
java.sql.SQLException: com.mchange.v2.c3p0.ComboPooledDataSource [...] has been closed() -- you can no longer use it.

The above shows a problem with the Quartz Scheduler, which can no longer access the database. The same problem happens e.g. when interacting with database-backed repositories.

I use:

• Jetty 9.1.0.v20131115
• C3P0 0.9.1.1 ... (I also tried 0.9.5-pre6, same symptoms)
• MYSQL J/Connector 5.1.27

The JNDI DataSource is set up like this in my Jetty XML configuration:

<Configure id="Server" class="org.eclipse.jetty.server.Server">
  <New id="jdbc-mydb" class="org.eclipse.jetty.plus.jndi.Resource">
    <Arg></Arg>
    <Arg>jdbc/mydb</Arg>
    <Arg>
      <New class="com.mchange.v2.c3p0.ComboPooledDataSource">
        <Set name="DriverClass">com.mysql.jdbc.Driver</Set>
        <Set name="JdbcUrl">jdbc:mysql://localhost:3306/mydb?useUnicode=true&amp;characterEncoding=UTF-8</Set>
        <Set name="User">user</Set>
        <Set name="Password">pass</Set>
        <Set name="MaxPoolSize">40</Set>
        <Set name="MinPoolSize">20</Set>
        <Set name="MaxIdleTime">28000</Set>
      </New>
    </Arg>
  </New>
</Configure>

When instead of C3P0 I use a plain MySQL driver, then my servlets deploy, undeploy, and redeploy without problems:

<Configure id="Server" class="org.eclipse.jetty.server.Server">
  <New id="jdbc-mydb" class="org.eclipse.jetty.plus.jndi.Resource">
    <Arg></Arg>
    <Arg>jdbc/mydb</Arg>
    <Arg>
      <New class="com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource">
        <Set name="Url">jdbc:mysql://localhost:3306/mydb?useUnicode=true&amp;characterEncoding=UTF-8</Set>
        <Set name="User">user</Set>
        <Set name="Password">pass</Set>
      </New>
    </Arg>
  </New>
</Configure>

How can undeploy a servlet without "closing" the C3P0 DataSource?

Answer 1

For my case, the symptoms turned out to be caused by neither C3P0 nor Jetty: The JNDI-shared C3P0 DataSource was closed by Spring at the servlet's Spring container shutdown time (i.e. when issuing rm -f ${jetty.base}/webapps/c.war).

Explanation
My servlets are Spring-based. I exploit Spring's JavaConfig. In my Spring-based servlets, I use @Configuration classes such as this one:

@Configuration
public class MainConfig {
    //...
    @Bean
    DataSource dataSource() {
        DataSource myds = null;
        JndiTemplate jndi = new JndiTemplate();
        try {
            myds = (DataSource) jndi.lookup("java:comp/env/jdbc/mydb");
        } catch (NamingException e) {
            logger.error("NamingException for java:comp/env/jdbc/mydb", e);
        }
        return myds;
    }
}

Spring's @Bean has the following relevant semantics:

As a convenience to the user, the container will attempt to infer a destroy method against object returned from the @Bean method. [...] This 'destroy method inference' is currently limited to detecting only public, no-arg methods named close. The method may be declared at any level of the inheritance hierarchy, and will be detected regardless of the return type of the @Bean method, i.e. detection occurs reflectively against the bean instance itself at creation time.

To disable destroy method inference for a particular @Bean, specify an empty string as the value, e.g. @Bean(destroyMethod="").

It turns out that com.mchange.v2.c3p0.ComboPooledDataSource does have a close() method (in its supertype com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource).

So when undeployment of my servlet takes place, at Spring container shutdown time, Spring will call ComboPooledDataSource.close(). IMHO, semantically, in this particular situation, Spring is wrong to call this method.

The solution to my cause is to annotate my DataSource bean with @Bean(destroyMethod=""):

@Configuration
public class MainConfig {
    //...
    @Bean(destroyMethod="")
    DataSource dataSource() {
        DataSource myds = null;
        JndiTemplate jndi = new JndiTemplate();
        try {
            myds = (DataSource) jndi.lookup("java:comp/env/jdbc/mydb");
        } catch (NamingException e) {
            logger.error("NamingException for java:comp/env/jdbc/mydb", e);
        }
        return myds;
    }
}

Now all my servlets get gracefully undeployed, redployed, and deployed.