Hazelcast class not found exception on EntryExpiredListener

Question

I am getting a serialization exception due to class not found when using an EntryExpiredListener.

I created a test jar and narrowed the issue down.

Test jar contains TestService, CacheKey and CacheValue. TestService inserts key type CacheKey and value type CacheValue to IMap. It registers an EntryExpiryListener to listen for the expiry of the inserted entries.

TestService code is simple:

hazelcastService.put(
        new CacheKey("11", "12345",  "678910", 1624174140000L),
        new CacheValue(new HashMap<>())); // ttl is set to 30 sec

Collection<HazelcastInstance> instances = HazelcastClient.getAllHazelcastClients();
Optional<HazelcastInstance> instance = instances.stream().findAny();
if (!instance.isPresent()) {
    throw new IllegalStateException("Can not find Hazelcast instance!");
}
// HazelcastInstance is created by framework component
IMap<CacheKey, CacheValue> cacheMap = instance.get().getMap(cacheName);

cacheMap.addEntryListener((EntryExpiredListener<CacheKey, CacheValue>) event -> {
    CacheKey key = event.getKey();
    CacheValue val = event.getOldValue();
}, true);

public class CacheKey implements Serializable {
    private static final long serialVersionUID = 123412453256L;
    private final String type;
    private final String field1;
    private final String field2;
    private final long timestamp;
    
    // constructor, getters

public class CacheValue implements Serializable {
    private static final long serialVersionUID = 658933453256L;
    private final HashMap<String, Object> recordMap;
    
    // constructor, getters

Issue

When entry expires, it throws serialization exception due to class not found exception for CacheKey.

Method threw 'com.hazelcast.nio.serialization.HazelcastSerializationException' exception. Cannot evaluate com.hazelcast.map.impl.DataAwareEntryEvent.toString()

2021-10-19 16:41:13,921 ERROR [hz.client_1.event-4] c.h.c.impl.spi.ClientListenerService hz.client_1 [nifi] [4.2] hz.client_1.event-4 caught an exception while processing:com.hazelcast.client.impl.spi.impl.listener.ClientListenerServiceImpl$ClientEventProcessor@782e8844
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: com.myapp.expirytest.CacheKey
        at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:90)
        at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:79)
        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:208)
        at com.hazelcast.map.impl.DataAwareEntryEvent.getKey(DataAwareEntryEvent.java:74)
        at com.ipfli.drs.nifi.processor.hzexpirytest.ExpiryTest.lambda$onScheduled$0(ExpiryTest.java:112)
        at com.hazelcast.map.impl.MapListenerAdaptors.lambda$null$6(MapListenerAdaptors.java:93)
        at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)
        at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)
        at com.hazelcast.client.impl.proxy.ClientMapProxy$AbstractClientMapEventHandler.handleEntryEvent(ClientMapProxy.java:2004)
        at com.hazelcast.client.impl.proxy.ClientMapProxy$ClientMapEventHandler$1.handleEntryEvent(ClientMapProxy.java:1981)
        at com.hazelcast.client.impl.protocol.codec.MapAddEntryListenerCodec$AbstractEventHandler.handle(MapAddEntryListenerCodec.java:164)
        at com.hazelcast.client.impl.proxy.ClientMapProxy$ClientMapEventHandler.handle(ClientMapProxy.java:1989)
        at com.hazelcast.client.impl.proxy.ClientMapProxy$ClientMapEventHandler.handle(ClientMapProxy.java:1971)
        at com.hazelcast.client.impl.spi.impl.listener.ClientListenerServiceImpl.handleEventMessageOnCallingThread(ClientListenerServiceImpl.java:189)
        at com.hazelcast.client.impl.spi.impl.listener.ClientListenerServiceImpl$ClientEventProcessor.run(ClientListenerServiceImpl.java:356)
        at com.hazelcast.internal.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:245)
        at com.hazelcast.internal.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:228)
Caused by: java.lang.ClassNotFoundException: com.myapp.expirytest.CacheKey
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at com.hazelcast.internal.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:289)
        at com.hazelcast.internal.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:249)
        at com.hazelcast.internal.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:910)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1984)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1848)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2158)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1665)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:501)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:459)
        at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:86)
        ... 17 common frames omitted

Test jar is deployed to user-lib folder. I know CacheKey class is deployed server side properly because I use a different service to fetch it, it works without issues. Done many other tests. Only issue is with EntryExpiredListener.

Hazelcast server and JVM client are both running on my local machine, single node. Hazelcast version 4.2.1

So everything is in the same jar and I know class is deployed server side. What am I missing?

Answer 1

Solved this by setting class loader for HazelcastClient:

ClientConfig clientConfig = new ClientConfig();
clientConfig.setClassLoader(TestService.class.getClassLoader());

I debugged Hazelcast code and noticed it's trying to use NarClassLoader to load the class. NarClassLoader comes from the Apache Nifi framework I am using. Nifi packages code into .nar files instead of .jar files.

No idea why it would try to use this class loader just for EntryExpiredListener. Other services are working meaning it's using a different class loader and loading the class successfully.