Main thread locks object and blocks HZ thread on Spring startup

Question

I have a weird problem with my Spring application with Hazelcast caching on applicatin startup

When some beans are creating the main thread has to get some items from HZ map, then HZ thread tries to lock the already locked by main thread object and becomes blocked.

So the Spring config would be

<bean id="holder" class="org.example.api.Holder">
    <constructor-arg>
        <bean factory-bean="cachedRepository" factory-method="getAllValueIds">
            <constructor-arg>
                <util:constant static-field="org.example.api.domain.MyStaticField.SOME_FIELD"/>
            </constructor-arg>
        </bean>
    </constructor-arg>
</bean>

<bean id="cachedRepository" class="org.example.api.repository.CachedRepository">
    <property name="byIdCache">
        <bean factory-bean="hazelcastInstance" factory-method="getMap">
            <constructor-arg value="byId"/>
        </bean>
    </property>
</bean>

on CachedRepository I'd have

public List<Integer> getAllValueIds(SomeField someField) {
    return byIdCache.get(someField);
}

here I have a Holder bean that should be initialized with a list of Integers from cachedRepository, which will get it from the byIdCache from HZ using map loader

main thread dump:

"org.example.Launcher.main()" #25 prio=5 os_prio=0 tid=0x000001b6ef442800 nid=0x578 waiting on condition [0x0000000b5f0f9000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:160)
    at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:434)
    at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:351)
    at com.hazelcast.map.impl.proxy.NearCachedMapProxyImpl.getInternal(NearCachedMapProxyImpl.java:115)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:120)
    at org.example.api.repository.CachedRepository.getById(CachedRepository.java:21)
    at org.example.api.repository.CachedRepository.getAllValueIds(CachedRepository.java:43)
    at org.example.api.Holder.init(Holder.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1763)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1700)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1630)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    - locked <0x00000006ca38d540> (a java.util.concurrent.ConcurrentHashMap)

HZ thread dump:

"hz.application.partition-operation.thread-4" #89 prio=5 os_prio=0 tid=0x000001b6ed53e000 nid=0x32a4 waiting for monitor entry [0x0000000b637fe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:188)
    - waiting to lock <0x00000006ca38d540> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:492)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389)
    at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:170)
    at org.springframework.beans.factory.annotation.BeanFactoryAnnotationUtils.qualifiedBeanOfType(BeanFactoryAnnotationUtils.java:89)
    at org.springframework.beans.factory.annotation.BeanFactoryAnnotationUtils.qualifiedBeanOfType(BeanFactoryAnnotationUtils.java:66)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.determineQualifiedTransactionManager(TransactionAspectSupport.java:396)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.determineTransactionManager(TransactionAspectSupport.java:377)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:272)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
    at org.example.api.cache.MapLoader$$EnhancerBySpringCGLIB$$d59bbb21.load(<generated>)
    at com.hazelcast.map.impl.MapStoreWrapper.load(MapStoreWrapper.java:165)
    at com.hazelcast.map.impl.mapstore.writethrough.WriteThroughStore.load(WriteThroughStore.java:72)
    at com.hazelcast.map.impl.mapstore.writethrough.WriteThroughStore.load(WriteThroughStore.java:28)
    at com.hazelcast.map.impl.recordstore.DefaultRecordStore.loadRecordOrNull(DefaultRecordStore.java:325)
    at com.hazelcast.map.impl.recordstore.DefaultRecordStore.get(DefaultRecordStore.java:513)
    at com.hazelcast.map.impl.operation.GetOperation.run(GetOperation.java:41)
    at com.hazelcast.spi.Operation.call(Operation.java:170)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:208)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:197)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:147)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:125)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:110)

you can see that main thread locked the object

- locked <0x00000006ca38d540> (a java.util.concurrent.ConcurrentHashMap)

and then HZ one blocked waiting to lock it too

"hz.application.partition-operation.thread-4" #89 prio=5 os_prio=0 tid=0x000001b6ed53e000 nid=0x32a4 waiting for monitor entry [0x0000000b637fe000] java.lang.Thread.State: BLOCKED (on object monitor) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:188) - waiting to lock <0x00000006ca38d540> (a java.util.concurrent.ConcurrentHashMap)

This same setup works perfectly well on Spring v4.1.3.RELEASE, but is broken on Spring v4.1.4 and forth (cannot check Spring 5 or 6) Hazelcast version is 3.11 in both cases, so I assume the breaking change would be in Spring 4.1.4 somewhere. I can go up to 4.3.30.RELEASE with Spring version, but no further.

Any ideas how to fix this on Spring 4.1.4-4.3.30 are highly appreciated.

Answer 1

You should not call the getAllValueIds method inside the init method of the Holder service. Spring cannot provide an instance of the Holder service due to issues in transaction management.

Instead, you should add a method to your Holder service, such as getCachedIds. Within this method, call CachedRepository.getAllValueIds rather than doing so in the init method.

This way, you will be able to call the getCachedIds service as follows:

@Configuration
@ImportResource("classpath:beans.xml")
public class SpringBootXmlApplication implements CommandLineRunner {

    @Autowired 
    private Pojo pojo;

    public static void main(String[] args) {
        SpringApplication.run(SpringBootXmlApplication.class, args);
    }

  @Bean
    public CommandLineRunner commandLineRunner(ApplicationContext ctx) {
        return args -> {
            
            Holder holderService = ctx.getBean(Holder.class);
            holderService.getCachedIds();
        };
    }

}