HazelcastSerializationException with Hazelcast 4.2 and bucket4j 6.2.0

Question

I have upgraded my Hazelcast and bucket 4j. And now I have error com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: io.github.bucket4j.grid.CommandResult

What is strange - this error happens only when I run my application on Kubernetes. When I run it on virtual machine or on my latop everything works OK

My application is using Spring Boot 2.2.1 and Spring Cloud Gateway and react

What I am doing wrong ?

My gradle dependencies

  implementation "com.github.vladimir-bukhtoyarov:bucket4j-core:6.2.0"
   implementation "com.github.vladimir-bukhtoyarov:bucket4j-jcache:6.2.0"
   implementation "com.github.vladimir-bukhtoyarov:bucket4j-hazelcast:6.2.0"
   implementation "javax.cache:cache-api:1.0.0"
   implementation "com.hazelcast:hazelcast:4.2"
   implementation "com.hazelcast:hazelcast-eureka-one:2.0.1"

My Hazelcast configuration code

 ```
    Config config = new Config();
   config.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(true);
        config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
        config.getNetworkConfig().getJoin().getAwsConfig().setEnabled(false);
        config.getNetworkConfig().getJoin().getEurekaConfig().setEnabled(false);
        config.getNetworkConfig().getJoin().getKubernetesConfig().setEnabled(false);

        if (eurekaClientOption.isPresent()) {
            EurekaClient eurekaClient = eurekaClientOption.get();
            String appName = eurekaClient.getApplicationInfoManager().getEurekaInstanceConfig().getAppname();
            logger.info("hazelcast appName :" + appName);
            Application application = eurekaClient.getApplication(appName);
            if (application != null) {
                application.getInstancesAsIsFromEureka().forEach(
                        instanceInfo -> {
                            config.getNetworkConfig().getJoin().getTcpIpConfig().addMember(instanceInfo.getIPAddr() + ":" + port);
                            logger.info("hazelcast TcpIpConfig : {}",config.getNetworkConfig().getJoin().getTcpIpConfig());
                        }
                );
            } else {
                logger.error("hazelcast empty eureka application [{}]",appName);
            }
        } else {
            logger.error("hazelcast empty eurekaClient");
        }
        config.getNetworkConfig().setPort(port).setPortAutoIncrement(false);
        config.setLiteMember(false);
        logger.info("hazelcast config {}",config);
    config.setProperty("hazelcast.phone.home.enabled", "false");
    config.setProperty("hazelcast.initial.min.cluster.size","1");
    config.setProperty("hazelcast.socket.server.bind.any", "false");
   HazelcastInstance hazelcastInstance = new HazelcastInstanceFactory(config).getHazelcastInstance();

My bucket4j code

    ```
    if (hazelcastInstance != null) {
            IMap map = hazelcastInstance.getMap("bucket");
            int initialCapacity = (throttlingRule.getBandwith() + throttlingRule.getOverdraftBandwith());
            this.bucket = Bucket4j.extension(Hazelcast.class)
                    .builder()
                    .addLimit(Bandwidth.classic(initialCapacity, Refill.smooth(throttlingRule.getBandwith(), Duration.of(throttlingRule.getDuration(), throttlingRule.getDurationUnit()))))
                    .build(map, key, RecoveryStrategy.RECONSTRUCT);

            VerboseResult<EstimationProbe> verboseResult = bucket.asVerbose().estimateAbilityToConsume(1);
            BucketConfiguration bucketConfiguration = verboseResult.getConfiguration();
            logger.info("bucket for rule [{}] and key [{}] configuration [{}] availableTokens [{}]",throttlingRule,key,bucketConfiguration,bucket.getAvailableTokens());
        } else  {
            logger.warn("hazelcastInstance == null");
        }

  public ConsumptionProbe throttleProbe() {
       assert bucket != null;
       CompletableFuture<ConsumptionProbe> probe = bucket.asAsync().tryConsumeAndReturnRemaining(1);
       return probe.join();
   }

Exception

j.l.ClassNotFoundException: io.github.bucket4j.grid.CommandResult
    at j.i.l.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at j.i.l.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at j.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at c.h.i.n.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:289)
    at c.h.i.n.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:249)
    at c.h.i.n.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:910)
    at j.i.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1995)
    at j.i.ObjectInputStream.readClassDesc(ObjectInputStream.java:1862)
    at j.i.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2169)
    at j.i.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
    at j.i.ObjectInputStream.readObject(ObjectInputStream.java:493)
    at j.i.ObjectInputStream.readObject(ObjectInputStream.java:451)
    at c.h.i.s.i.d.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:86)
    ... 11 common frames omitted
Wrapped by: c.h.n.s.HazelcastSerializationException: java.lang.ClassNotFoundException: io.github.bucket4j.grid.CommandResult
    at c.h.i.s.i.d.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:90)
    at c.h.i.s.i.d.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:79)
    at c.h.i.s.i.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
    at c.h.i.s.i.AbstractSerializationService.toObject(AbstractSerializationService.java:208)
    at c.h.s.i.DelegatingCompletableFuture$DeserializingFunction.apply(DelegatingCompletableFuture.java:518)
    at c.h.s.i.AbstractInvocationFuture.lambda$unblockCompose$5(AbstractInvocationFuture.java:953)
    at j.u.c.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
    at j.u.c.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at j.u.c.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at j.u.c.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at j.u.c.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at j.u.c.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Wrapped by: j.u.c.CompletionException: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: io.github.bucket4j.grid.CommandResult
    at j.u.c.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
    at j.u.c.CompletableFuture.completeThrowable(CompletableFuture.java:346)
    at j.u.c.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
    at j.u.c.CompletableFuture.unipush(CompletableFuture.java:589)
    at j.u.c.CompletableFuture.uniApplyStage(CompletableFuture.java:660)
    at j.u.c.CompletableFuture.thenApplyAsync(CompletableFuture.java:2104)
    at c.h.s.i.InternalCompletableFuture.thenApply(InternalCompletableFuture.java:78)
    at i.g.b.AbstractBucket$1.tryConsumeAndReturnRemaining(AbstractBucket.java:147)
    at p.b.g.t.ThrottlingRuleBucket.throttleProbe(ThrottlingRuleBucket.java:64)

Answer 1

@Jacuo I was facing the same issue where my application on my laptop was working fine but from within Docker container it was failing with HazelcastSerializationException.

I found out that the issue was not with Serialization on Hazelcast member side but in fact issue was on client side. Following change fixed the issue for me.

new ClientConfig()
     .setClassLoader(CommandResult.class.getClassLoader())  <------ THIS LINE HERE
     .setClusterName(clusterName)
     .setInstanceName(String.format("%s-%s", clusterName, UUID.randomUUID()))
     .setUserCodeDeploymentConfig(userCodeDeploymentConfig)
     .setNetworkConfig(networkConfig);

For more information see https://github.com/vladimir-bukhtoyarov/bucket4j/issues/162#issuecomment-966890797

Answer 2

The classes need to be present in the cluster members. Can you confirm that the dependencies you mention included in your deployment? If they are not on the member image, you can configure user-code-deployment (https://docs.hazelcast.com/imdg/latest/clusters/deploying-code-on-member.html) to allow a member to load classes from a client.