Facing Infinispan Timeout Exceptions and PostgreSQL Connection Closure: Need Help Resolving DB Communication Issues

Question

I have keycloak 20.0.5 with infinispan 13 running in k8 environment having 3 pods also with postgresql database. I have enabled quarkus connection pooling and keycloak health check and matrix are also enabled.

Somehow i faced postgresql failover which causes DB connection interruptions after that keycloak becomes unstable, and started giving following exceptions.

failover was just for around 200 milliseconds.

although after 3 hours it becomes stable working as expected. but 3 hours is too much in production environment.

also in 3 hours keycloak pods restarts multiple times, for stability.

Following are the logs.

"2024-02-07 04:14:18,685 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command GetKeyValueCommand on Cache 'sessions', writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3220489 from sso-keycloak-2-33708,sso-keycloak-0-45708
"2024-02-07 04:14:18,877 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command GetKeyValueCommand on Cache 'sessions', writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3220492 from sso-keycloak-2-33708,sso-keycloak-0-45708
"2024-02-07 04:14:21,146 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command GetKeyValueCommand on Cache 'sessions', writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3227047 from sso-keycloak-0-45708,sso-keycloak-1-60550
"2024-02-07 04:14:25,737 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p4-t1) ISPN000136: Error executing command GetKeyValueCommand on Cache 'sessions', writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 3294318 from sso-keycloak-1-60550,sso-keycloak-2-33708
"2024-02-07 04:14:38,314 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t3) ISPN008001: Failed clearing cache store: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:38,314 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t3) ISPN008032: Error during rollback of JDBC transaction: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:38,396 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t2) ISPN008001: Failed clearing cache store: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:38,396 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t2) ISPN008032: Error during rollback of JDBC transaction: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:38,711 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t3) ISPN008001: Failed clearing cache store: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:38,711 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t3) ISPN008032: Error during rollback of JDBC transaction: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:45,979 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t6) ISPN008001: Failed clearing cache store: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:45,980 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t6) ISPN008032: Error during rollback of JDBC transaction: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:46,186 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t2) ISPN008001: Failed clearing cache store: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:46,187 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t2) ISPN008032: Error during rollback of JDBC transaction: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:49,045 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t3) ISPN008001: Failed clearing cache store: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:14:49,045 ERROR [org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore] (blocking-thread--p3-t3) ISPN008032: Error during rollback of JDBC transaction: org.postgresql.util.PSQLException: Connection has been closed.
"2024-02-07 04:15:26,308 ERROR [org.infinispan.PERSISTENCE] (blocking-thread--p3-t5) ISPN008027: SQL error while fetching stored entry with key: uuid, lockingKey: uuid: org.postgresql.util.PSQLException: Connection has been closed.

I want to minimize the downtime, like If DB goes down due to any reason and ASAP becomes available, Keycloak should become stable ASAP.