HikariCP connection pool not shutting down

Question

Problem description

I am running Unit Tests in a Play Framework application. After a couple tests, they start failing and the reason why is that the PostgreSQL limit of 100 open connections is reached. Each Unit Test has a connection pool of 10, so after 10 Unit tests, the fails start. Keep in mind, the tests run very fast and the pool is created and shut down several times in less than a second. Using Debug Logging, I don't see any unexpected behavior (or I haven't enabled all of the debug informations).

Managing the connection pool

I am not managing the connection pool myself (it's managed by the Play Framework) and I can see in the log that the pool is shut down after each Unit Test.

[info] application - Creating Pool for datasource 'default'
[info] p.a.d.HikariCPConnectionPool - datasource [default] bound to JNDI as DefaultDS
[info] application - ApplicationTimer demo: Starting application at 2017-04-20T08:50:46.096Z
[info] application - ApplicationTimer demo: Stopping application at 2017-04-20T08:50:46.129Z after 1s.
[info] application - Shutting down connection pool.

In the graph of the database you can see how the pool starts the connections after every unit test and suddenly after the fails start, the open connections go down instantanly. I don't see any connection shutting down between the tests.

Example Unit Test

Because I couldn't find any good examples of Unit Tests in the style I am doing, I can not confirm that my Unit Tests are flawless (in the sense of: I am using the provided tools of Play Framework correctly). Maybe you can spot some misusage of the framework itself that would prevent the Connection Pool to be shutdown.

public class UserPersistenceTesting {
    @Inject
    private JPAApi jpaApi;

    @Before
    public void setup() {
        GuiceApplicationBuilder builder = new GuiceApplicationLoader()
                .builder(new ApplicationLoader.Context(Environment.simple()));
        Guice.createInjector(builder.applicationModule()).injectMembers(this);
    }

    @After
    public void teardown() {
        jpaApi.shutdown();
    }

    @Test
    public void testRegisterNewUser() {
        this.jpaApi.withTransaction(() -> {
            UsersRepository repository = new UsersRepositoryJPA(jpaApi);
            Date creationDate = new Date();
            User expectedUser = new User(null, "max.muster@hsr.ch", "Muster", "Max", creationDate, Gender.MALE);
            repository.registerUser(expectedUser);

            User actualUser = repository.getUserById(expectedUser.getId());

            assertEquals(expectedUser, actualUser);
        });
    }
}

What I think is the problem

I have the feeling that there is a delay between the "shutdown" of the pool and the releasing of the connections. Because the Unit Tests run so fast, the pool would not be able to be released before the Database starts throwing exceptions.

What I have ruled out

Here is a list of errors that are ruled out:

More than 10 concurrent tests run at the same time and fill the connection pool. -> I have disabled concurrent tests and the errors still occurr.

Occurring exceptions

[error] o.p.Driver - Connection error:
org.postgresql.util.PSQLException: FATAL: tut mir leid, schon zu viele Verbindungen
  Ort: Datei: proc.c, Routine: InitProcess, Zeile: 340.
  Server SQLState: 53300
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.(PgConnection.java:194)
        at org.postgresql.Driver.makeConnection(Driver.java:431)
        at org.postgresql.Driver.connect(Driver.java:247)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
        at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:506)
[error] Test integration.database.EventPersictenceTesting.testRegisterNewEvent failed: play.api.Configuration$$anon$1: Configuration error[Cannot connect to database [default]], took 0.094 sec
[error]     at play.api.Configuration$.configError(Configuration.scala:155)
[error]     at play.api.Configuration.reportError(Configuration.scala:808)
[error]     at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:48)
[error]     at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:42)
[error]     at scala.collection.immutable.List.foreach(List.scala:381)
[error]     at play.api.db.DefaultDBApi.connect(DefaultDBApi.scala:42)
[error]     at play.api.db.DBApiProvider.get$lzycompute(DBModule.scala:72)
[error]     at play.api.db.DBApiProvider.get(DBModule.scala:62)
[error]     at play.api.db.DBApiProvider.get(DBModule.scala:58)
[error]     at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
[error]     at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
[error]     at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
[error]     at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
[error]     at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
[error]     at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
[error]     at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
[error]     at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
[error]     at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
[error]     at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
[error]     at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
[error]     at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
[error]     at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
[error]     at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
[error]     at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
[error]     at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54)
[error]     at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
[error]     at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:93)
[error]     at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:80)
[error]     at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
[error]     at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:80)
[error]     at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:62)
[error]     at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:984)
[error]     at com.google.inject.util.Providers$GuicifiedProviderWithDependencies.initialize(Providers.java:149)
[error]     at com.google.inject.util.Providers$GuicifiedProviderWithDependencies$$FastClassByGuice$$2a7177aa.invoke()
[error]     at com.google.inject.internal.cglib.reflect.$FastMethod.invoke(FastMethod.java:53)
[error]     at com.google.inject.internal.SingleMethodInjector$1.invoke(SingleMethodInjector.java:57)
[error]     at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:91)
[error]     at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
[error]     at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:93)
[error]     at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:80)
[error]     at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
[error]     at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:80)
[error]     at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:174)
[error]     at com.google.inject.internal.Initializer.injectAll(Initializer.java:108)
[error]     at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:174)
[error]     at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
[error]     at com.google.inject.Guice.createInjector(Guice.java:96)
[error]     at com.google.inject.Guice.createInjector(Guice.java:84)
[error]     at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:181)
[error]     at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:123)
[error]     at play.api.test.FakeApplication.(Fakes.scala:209)
[error]     at play.test.FakeApplication.(FakeApplication.java:51)
[error]     at play.test.Helpers.fakeApplication(Helpers.java:124)
[error]     at play.test.WithApplication.provideFakeApplication(WithApplication.java:46)
[error]     at play.test.WithApplication.provideApplication(WithApplication.java:33)
[error]     at play.test.WithApplication.startPlay(WithApplication.java:51)
[error]     ...
[error] Caused by: Configuration error: Configuration error[Failed to initialize pool: FATAL: tut mir leid, schon zu viele Verbindungen
[error]   Ort: Datei: proc.c, Routine: InitProcess, Zeile: 340.
[error]   Server SQLState: 53300]
[error]     at play.api.Configuration$.configError(Configuration.scala:155)
[error]     at play.api.PlayConfig.reportError(Configuration.scala:999)
[error]     at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:70)
[error]     at play.api.db.PooledDatabase.createDataSource(Databases.scala:199)
[error]     at play.api.db.DefaultDatabase.dataSource$lzycompute(Databases.scala:123)
[error]     at play.api.db.DefaultDatabase.dataSource(Databases.scala:121)
[error]     at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
[error]     at play.api.db.DefaultDatabase.getConnection(Databases.scala:138)
[error]     at play.api.db.DefaultDBApi$$anonfun$connect$1.apply(DefaultDBApi.scala:44)
[error]     ... 88 more
[error] Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: tut mir leid, schon zu viele Verbindungen
[error]   Ort: Datei: proc.c, Routine: InitProcess, Zeile: 340.
[error]   Server SQLState: 53300
[error]     at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:512)
[error]     at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:105)
[error]     at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:71)
[error]     at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:58)
[error]     at play.api.db.HikariCPConnectionPool$$anonfun$1.apply(HikariCPModule.scala:54)
[error]     at scala.util.Try$.apply(Try.scala:192)
[error]     at play.api.db.HikariCPConnectionPool.create(HikariCPModule.scala:54)
[error]     ... 94 more
[error] Caused by: org.postgresql.util.PSQLException: FATAL: tut mir leid, schon zu viele Verbindungen
[error]   Ort: Datei: proc.c, Routine: InitProcess, Zeile: 340.
[error]   Server SQLState: 53300
[error]     at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
[error]     at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
[error]     at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
[error]     at org.postgresql.jdbc.PgConnection.(PgConnection.java:194)
[error]     at org.postgresql.Driver.makeConnection(Driver.java:431)
[error]     at org.postgresql.Driver.connect(Driver.java:247)
[error]     at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
[error]     at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
[error]     at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
[error]     at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:506)
[error]     ... 100 more
[error] Failed: Total 18, Failed 6, Errors 0, Passed 12
[error] Failed tests:
[error]         integration.database.EventPersictenceTesting
[error]         integration.rest.IntegrationTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 31 s, completed 20.04.2017 10:50:52