Are spring-data-redis connections not properly released when transaction support is enabled?

Question

In our Spring 4 project we would like to have database transactions that involve Redis and Hibernate. Whenever Hibernate fails, for example due to optimistic locking, the Redis transaction should be aborted as well.

This seems to work for

1. Single-threaded transaction execution.
2. Multi-threaded transaction execution, as long as the transaction only includes a single Redis call.
3. Multi-threaded transaction execution with multiple Redis calls, if Hibernate is excluded from our configuration.

As soon as a transaction includes multiple Redis calls, and Hibernate is configured to take part in the transactions, there seems to be a problem with connection binding and multithreading. Threads are stuck at RedisConnectionUtils.bindConnection(), probably since the JedisPool runs out of connections.

This can be reproduced as follows.

@Service
public class TransactionalService {

    @Autowired
    @Qualifier("redisTemplate")
    private RedisTemplate<String, Object> redisTemplate;

    @Transactional
    public void processTask(int i){

        redisTemplate.convertAndSend("testChannel", new Message());
        redisTemplate.convertAndSend("testChannel", new Message());
    }
}

We use a ThreadPoolTaskExecutor having a core pool size of 50 to simulate multithreaded transactions.

@Service
public class TaskRunnerService {

    @Autowired
    private TaskExecutor taskExecutor;

    @Autowired
    private TransactionalService transactionalService;

    public void runTasks() {

        for (int i = 0; i < 100; i++) {

            final int j = i;

            taskExecutor.execute(new Runnable() {

                @Override
                public void run() {
                    transactionalService.processTask(j);
                }
            });
        }
    }
}

Running this results in all taskExecutor threads hanging in JedisPool.getResource():

  "taskExecutor-1" - Thread t@18
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <1b83c92c> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:524)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
    at redis.clients.util.Pool.getResource(Pool.java:40)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:84)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:10)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)
    at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
    at org.springframework.data.redis.core.RedisConnectionUtils.bindConnection(RedisConnectionUtils.java:66)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:175)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
    at org.springframework.data.redis.core.RedisTemplate.convertAndSend(RedisTemplate.java:675)
    at test.TransactionalService.processTask(TransactionalService.java:23)
    at test.TransactionalService$$FastClassBySpringCGLIB$$9b3de279.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
  at test.TransactionalService$$EnhancerBySpringCGLIB$$a1b3ba03.processTask(<generated>)
  at test.TaskRunnerService$1.run(TaskRunnerService.java:28)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - locked <7d528cf7> (a java.util.concurrent.ThreadPoolExecutor$Worker)   

Redis Config

@Configuration
public class RedisConfig {

    @Bean
    public JedisConnectionFactory jedisConnectionFactory() {
        JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory();
        jedisConnectionFactory.setPoolConfig(new JedisPoolConfig());
        return jedisConnectionFactory;
    }

    @Bean
    public Jackson2JsonRedisSerializer<Object> jackson2JsonRedisSerializer() {
        Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new     Jackson2JsonRedisSerializer(Object.class);
        jackson2JsonRedisSerializer.setObjectMapper(objectMapper());
        return jackson2JsonRedisSerializer;
    }

    @Bean
    public StringRedisSerializer stringRedisSerializer() {
        return new StringRedisSerializer();
    }

    @Bean
    public RedisTemplate<String, Object> redisTemplate() {
        RedisTemplate<String, Object> redisTemplate = new RedisTemplate();
        redisTemplate.setConnectionFactory(jedisConnectionFactory());
        redisTemplate.setKeySerializer(stringRedisSerializer());
        redisTemplate.setValueSerializer(jackson2JsonRedisSerializer());
        redisTemplate.setEnableTransactionSupport(true);
        return redisTemplate;
    }

    @Bean
    public ObjectMapper objectMapper() {
        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
        objectMapper.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
        return objectMapper;
    }
}

Hibernate Config

@EnableTransactionManagement
@Configuration
public class HibernateConfig {

    @Bean
    public LocalContainerEntityManagerFactoryBean admin() {

        LocalContainerEntityManagerFactoryBean entityManagerFactoryBean = new     LocalContainerEntityManagerFactoryBean();
        entityManagerFactoryBean.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
        entityManagerFactoryBean.setPersistenceUnitName("test");

        return entityManagerFactoryBean;
    }

    @Bean
    public JpaTransactionManager transactionManager(
            @Qualifier("admin") LocalContainerEntityManagerFactoryBean     entityManagerFactoryBean) {

        JpaTransactionManager transactionManager = new JpaTransactionManager();
        transactionManager.setEntityManagerFactory(entityManagerFactoryBean.getObject());
        transactionManager.setDataSource(entityManagerFactoryBean.getDataSource());

        return transactionManager;
    }
}

Is this a bug in spring-data-redis or is something wrong in our configuration?

Answer 1

I had a very similar problem but bumping the maxTotal threads bothered me if the threads really weren't being released. Instead I had some code that rapidly did a get and then a set. I put this in a SessionCallback and it behaved much better. Hope that helps.

Answer 2

I found your question (coincidentally) right before I hit the exact same issue using opsForHAsh and putting many keys. A thread dump confirmed it.

What I found helped to get me going was to increase the thread pool in my JedisPoolConfig. I set it as follows, to 128, and that got me on my way again.

@Bean
JedisPoolConfig jedisPoolConfig() {
    JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
    jedisPoolConfig.setMaxTotal(128);
    return jedisPoolConfig;
}

I assume the pool was too small in my case, and all the threads were in use for my transaction, so were waiting indefinitely. Setting to total to 128 allowed me to continue. Try setting your config to a maxTotal that makes sense for your application.