Hibernate Connection Leaks

Question

I have a Java 7 web application stack that is deployed into a clustered environment using JBoss 7.1.3 and ActiveMQ 5.11.1. I use Hibernate 4.0.1 to connect our Java application to an RDS instance.

I have observed our environment appears to be leaking database connections. As we comb through our application, we are having trouble identifying where these additional database connections may be coming from. Our hibernate connections have relatively short life spans and are always closed inside a finally { } block.

Using queries on the database, I can see what the one node will stay constant at 20 connections and the other one (the master) will slowly grow in connection size. Eventually, this number grows so much we need to restart the application on that node which triggers fail over the other node. That node will then begin to grow in connections while the newly rebooted note will stay constant at 20.

Is it possible I am bumping into any known issues with Hibernate? Any suggestions on how to properly debug this?

Thanks.

Update 1

We are using a connection pool defined as follows:

<datasource jta="true" jndi-name="java:jboss/datasources/MySqlDS" pool-name="MySqlIDS" enabled="true" use-java-context="true" use-ccm="true">
                    <connection-url>jdbc:mysql://...</connection-url>
                    <driver>com.mysql</driver>
                    <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
                    <pool>
                        <min-pool-size>10</min-pool-size>
                        <max-pool-size>150</max-pool-size>
                        <prefill>true</prefill>
                        <use-strict-min>false</use-strict-min>
                        <flush-strategy>FailingConnectionOnly</flush-strategy>
                    </pool>
                    ...
                </datasource>

Update 2

Each time hibernate is used a forceCloseSessionInstance method is called which does the following.

private void forceCloseSessionInstance() {
        Session hsession = session.get();

if (hsession != null && hsession.isActive()  && !hsession.wasRolledBack()) {
            hsession.rollback();
        }

        for( ; ; ) {
            closeSessionInstance();
            if ( session.get() == null ) {
                return;
            }
        }
    }

public void closeSessionInstance() throws HibernateException {
        Integer level = transLevel.get();
        if (level == null) {
            transLevel.set(0);
            level = transLevel.get();
        }
        if (level > 0) {
            transLevel.set(level - 1);
        } else {
            Session s = session.get();
            session.set(null);
            if (s != null)
            {
                if (s.isOpen())
                {
                    s.flush();
                    s.clear();
                    s.close();
                }
            }
        }

    }

Answer 1

You use the finally block to indicate what you want to happen at the end of processing, regardless of what happens within the try-block. But if you put multiple things in your finally block, then you risk having things in the finally block not complete if something earlier throws an exception. Here's a common way to write JDBC code that can cause a connection leak:

try {
    ...
} finally {
    resultSet.close();
    statement.close();
    connection.close();
}

If either the resultSet or statement throws an exception when close is called (due to, say, the jdbc object trying to tell the database server that it can free resources that it has allocated for that object but a network hiccup occurs), then the connection doesn't get closed.

The commonality with your posted code is there is a lot going on in your cleanup, and if anything goes wrong with any of it then the session doesn't get closed.

The solution is to restrict the scope for exceptions thrown by one thing to keep other things from being called, for instance by nesting finally blocks:

try {
    try {
        try {
            ...
        } finally {
            try {
                resultSet.close();
            } catch (SQLException e) {
                log.info(e);
            }
        }
    } finally {
        try {
            statement.close();
        } catch (SQLException e) {
            log.info(e);
        }
    }
} finally {
    try {    
        connection.close();
    } catch (SQLException e) {
        log.info(e);
    }
}

so if anything goes wrong with the inner finally blocks the outer ones still get called. This code also stops exceptions caused when closing resources from masking the exception thrown by the try-block (because the exception thrown by the try-block is the one with the useful information).

(Using try-with-resources works mostly the same way except that if the try block completes normally and any of the resources throw something, then the exception thrown on close does not get suppressed.)