How to prevent Liferay from closing external (non-Liferay) JNDI resources when redeploying modules?

Question

We have a Liferay Service Builder module (portlet) that uses a JDBC DataSource for an external (non-Liferay) database from a HikariCP database connection pool declared in Tomcat using JNDI.

When we first start the server, the JNDI connection works as expected (we're able to retrieve/edit database data), but when we try to redeploy (hot-deploy) the module, Liferay/Spring closes the connection pool during undeployment, so when the module deploys again, we receive errors in the Liferay log (see below) and the module is unable to perform any database operations (resulting in additional exceptions).

2024-10-04 18:41:33.620 ERROR [Refresh Thread: Equinox Container: ce2fdf2e-11b2-49cf-87b1-8d78cc1bcce5][DialectDetector:147] java.sql.SQLException: HikariDataSource HikariDataSource (appServices) has been closed.
java.sql.SQLException: HikariDataSource HikariDataSource (appServices) has been closed.
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:81)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy7.getMetaData(Unknown Source)
at com.liferay.portal.spring.hibernate.DialectDetector.getDialect(DialectDetector.java:56)
at com.liferay.portal.spring.hibernate.PortalHibernateConfiguration.newConfiguration(PortalHibernateConfiguration.java:144)
...

The only known resolution is to restart Tomcat after each module deployment, but this breaks our development workflow (hot-redeploy is a huge time saver) and our CI deployments.

How can we configure HikariCP/Tomcat/Liferay so that it doesn't close the HikariCP connection pools during module redeployment?

Installation / configuration / usage details

We're using Liferay 7.1.3-ga4.

We've "installed" and configured an "external" (non-Liferay) JNDI DataSource in the "global" Tomcat context using HikariCP as the database connection pool per the Liferay documentation at https://learn.liferay.com/w/dxp/installation-and-upgrades/installing-liferay/installing-liferay-on-an-application-server/setting-up-jndi-on-tomcat (also referencing https://liferay.dev/blogs/-/blogs/tomcat-hikaricp and https://github.com/brettwooldridge/HikariCP).

Specifically:

• HikariCP has been "installed" by moving hikaricp.jar and slf4j-api.jar from tomcat-9.0.91/webapps/ROOT/WEB-INF/lib to tomcat-9.0.91/lib/ext
• tomcat-9.0.91/conf/server.xml has been modified with new JNDI Resource definitions:

  <Server port="8005" shutdown="SHUTDOWN">
      ...
      <GlobalNamingResources>
          ...
          <Resource name="jdbc/appServices"
              factory="com.zaxxer.hikari.HikariJNDIFactory"
              type="javax.sql.DataSource"
              auth="Container"
              connectionTimeout="2000"
              idleTimeout="60000"
              maxLifetime="0"
              minimumIdle="5"
              maximumPoolSize="20"
              poolName="appServices"
              registerMbeans="true"
              leakDetectionThreshold="30000"
              dataSourceClassName="org.postgresql.ds.PGSimpleDataSource"
              dataSource.serverName="<host>"
              dataSource.databaseName="<database>"
              dataSource.portNumber="<port>"
              dataSource.user="<username>"
              dataSource.password="<password"
              dataSource.socketTimeout="45000" />
          ...
      </GlobalNamingResources>
      ...
  </Server>
  

• tomcat-9.0.91/conf/context.xml has been modified with ResourceLink definitions:

  <Context>
      ...
      <ResourceLink name="jdbc/appServices" global="jdbc/appServices" type="javax.sql.DataSource" />
      ...
  </Context>
  

Our Service Builder module is configured to use that JNDI DataSource via JDBC per the Liferay documentation at https://help.liferay.com/hc/en-us/articles/360017886812-Connecting-Service-Builder-to-External-Databases and then leverage the Service Builder APIs to perform operations on a model.

Specifically:

• app-services-portlet/app-services-portlet-service/src/main/resources/META-INF/spring/parent/ext-spring.xml has been added with Bean declarations to the DataSource:

  <?xml version="1.0" encoding="UTF-8"?>
  <beans 
      default-destroy-method="destroy"
      default-init-method="afterPropertiesSet"
      xmlns="http://www.springframework.org/schema/beans"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
  
      <bean class="com.liferay.portal.dao.jdbc.spring.DataSourceFactoryBean" id="liferayDataSourceFactory">
          <property name="propertyPrefix" value="app.db." />
          <property name="properties">
              <props>
                  <prop key="app.db.jndi.name">jdbc/appServices</prop>
              </props>
          </property>
      </bean>
  
      <bean class="org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy" id="liferayDataSource">
          <property name="targetDataSource" ref="liferayDataSourceFactory" />
      </bean>
  
      <alias alias="appDataSource" name="liferayDataSource" />
  </beans>
  

• app-services-portlet/app-services-portlet-service/service.xml has been updated with entities that use the DataSource specified in ext-spring.xml:

  <service-builder package-path="mil.psif.services" auto-namespace-tables="false">
      ...
      <entity name="WeatherConfig" table="weather_config" data-source="appDataSource" local-service="true" remote-service="true">
          <column name="id" db-name="id" primary="true" type="long"></column>
          <column name="portletId" db-name="portletid" type="String"></column>
          <column name="zipCode" db-name="zipcode" type="String"></column>
          <column name="longitude" db-name="longitude" type="double"></column>  
          <column name="latitude" db-name="latitude" type="double"></column>    
          <column name="createdBy" db-name="created_by" type="long"></column>
          <column name="createdDate" db-name="created_date" type="Date"></column>
          <column name="lastModifiedBy" db-name="last_modified_by" type="long"></column>
          <column name="lastModifiedDate" db-name="last_modified_date" type="Date"></column>
          <finder name="portletId" return-type="Collection">
              <finder-column name="portletId"></finder-column>
          </finder>
      </entity>
     ....
  </service-builder>
  

• app-services-portlet/app-services-portlet-services/src/main/java/com/app/services/services/impl/WeatherConfigLocalServiceImpl.java has been updated to perform actions using the Service Builder conventions:

  public class WeatherConfigLocalServiceImpl extends WeatherConfigLocalServiceBaseImpl {
      ...
      public WeatherConfig AddWeatherConfig(String portletId, String zipCode,
              double longitude, double latitude, long userId) {
          long resourceId = counterLocalService.increment(WeatherConfig.class.getName());
          WeatherConfig weatherConfig = weatherConfigPersistence.create(resourceId);
  
          weatherConfig.setPortletId(portletId);
          weatherConfig.setZipCode(zipCode);
          weatherConfig.setLongitude(longitude);
          weatherConfig.setLatitude(latitude);
          weatherConfig.setCreatedBy(userId);
          weatherConfig.setCreatedDate(new Date());
          weatherConfig.setLastModifiedBy(userId);
          weatherConfig.setLastModifiedDate(new Date());
  
          WeatherConfig savedWeatherConfig = weatherConfigPersistence.update(weatherConfig);
  
          return savedWeatherConfigHistory;
      }
      ...
  }
  

Troubleshooting and findings

As a recap from above, the use case is:

• Build and deploy the module using blade gw deploy
• Restart Tomcat/Liferay using blade server stop; blade server restart or restarting the Tomcat Windows service, etc.
  • Service performs "correctly" and operations successfully manipulate the database
• Build and redeploy the module (hot-redeploy) using blade gw deploy
  • Liferay/Spring (silently) closes the database connection pool during undeployment
  • During deployment of the updated module, many exceptions are thrown about the DataSource being closed
  • Service starts but fails all database operations and prints SQLExceptions to the logs
• Restart Tomcat/Liferay using blade server stop; blade server restart or restarting the Tomcat Windows service, etc.
  • Service starts and performs "correctly" and operations successfully manipulate the database

Some of the many articles I've read that seem related but don't actually solve my problem:

• https://github.com/brettwooldridge/HikariCP/issues/966
• Why does Datasources close on Tomcat 7 undeploy with Spring Boot
• Connection closes using JNDI in Tomcat
• Receiving Data source is Closed error using spring-boot 1.2.7 in Tomcat 8 Java 8
• db pool connections closed after war file is undeployed

Does not affect Tomcat DBCP nor pure JDBC Resources (only HikariCP)

Previously, we used (pure) JDBC Resources and Tomcat JDBC as a connection pool. The same use case (hot-redeploying) does not cause the Resources nor connection pool to stop, and the service continues to perform "correctly" after the redeployment.

JDBC Resource config in server.xml:

<Resource
    name="jdbc/appServices"
    auth="Container"
    type="javax.sql.DataSource"
    username="<user>"
    password="<password>"
    url="jdbc:postgresql://<host>:<port>/<database>" 
    driverClassName="org.postgresql.Driver"
    maxActive="20"
    maxIdle="10"
    maxWait="-1"
  />

Tomcat DBCP Resource config in server.xml:

<Resource
    name="jdbc/appServices"
    auth="Container"
    type="javax.sql.DataSource"
    driverClassName="org.postgresql.Driver"
    factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
    url="jdbc:postgresql://<host>:<port>/<database>"
    username="<username>"
    password="<password>"
    maxActive="20"
    maxIdle="5"
    maxWait="2000"
    removeAbandoned="true"
    removeAbandonedTimeout="45"
    testOnBorrow="true"
    validationQuery="SELECT 1"
    jdbcInterceptors="ResetAbandonedTimer"
  />

JMX indicates redeployment closes/destroys the Connection Pool

Using jconsole.exe to monitor the HikariCP connection pool shows that when the Service module is redeployed, the connection pool is destroyed.

Before redeployment:

After redeployment:

Again, when using Tomcat DBCP, jconsole shows that the connection pool remains running when the module is redeployed.

Live debugging indicates Liferay is closing the DataSources

When a breakpoint is set in com.zaxxer.hikari.HikariDataSource#close, the breakpoint is triggered when the Service module is redeployed, and the stack trace shows it is called by Liferay's com.liferay.portal.dao.jdbc.DataSourceFactoryImpl#destroyDataSource.

Notably, it appears that Spring attempts to clean up its beans when the module is undeployed, which calls Liferay's DataSourceFactoryImpl. When using HikariCP, the provided DataSource parameter is a com.zaxxer.hikari.HikariDataSource object, which implements java.io.Closable, and so the Liferay code calls its close method, which shuts down the DataSource (HikariCP pool). When using Tomcat DBCP, however, the provided DataSource parameter is a javax.sql.DataSource, which doesn't match any of the blocks in Liferay's DataSourceFactoryImpl code, and so Liferay doesn't close/shutdown the DataSource.

Final thoughts

You made it all the way here! You deserve a plaque!

Based on the above findings about Liferay code calling the DataSource destroy method, I'm hunting for some Liferay setting or trick to turn off it's kill-the-datasource-when-module-undeploys logic.

But, this feels like I must be missing something obvious, although I've been scouring the confusing and conflicting Liferay documentation and discussing with the Liferay Community Slack for almost a week now, and I'm still at a complete loss as to what is the "right" way to configure Liferay with external databases declared using JNDI and used in a Service Builder using JDBC.

So, please, any help would be greatly appreciated.