Spingboot latest release failing(In Cloud) with error java.lang.NoClassDefFoundError: java/net/http/HttpClient

Question

I was using SpringBoot3.2.2 and my project is working fine without any issue in Cloud environment. But recently i upgrade to 3.3.3 and checked with 3.3.4 but both places deployment is failing with following error

java.lang.NoClassDefFoundError: java/net/http/HttpClient at com.hazelcast.spi.utils.RestClient.(RestClient.java:87) at com.hazelcast.spi.utils.RestClient.createWithSSL(RestClient.java:110) at com.hazelcast.kubernetes.KubernetesClient.lambda$callGet$0(KubernetesClient.java:475) at com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:65) at com.hazelcast.kubernetes.KubernetesClient.callGet(KubernetesClient.java:473) at com.hazelcast.kubernetes.KubernetesClient.buildKubernetesApiUrlProvider(KubernetesClient.java:204) at com.hazelcast.kubernetes.KubernetesClient.(KubernetesClient.java:118) at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.buildKubernetesClient(KubernetesApiEndpointResolver.java:73) at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.(KubernetesApiEndpointResolver.java:49) at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.(HazelcastKubernetesDiscoveryStrategy.java:49) at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategyFactory.newDiscoveryStrategy(HazelcastKubernetesDiscoveryStrategyFactory.java:91) at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.buildDiscoveryStrategy(DefaultDiscoveryService.java:217) at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:163) at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.(DefaultDiscoveryService.java:58) at com.hazelcast.spi.discovery.impl.DefaultDiscoveryServiceProvider.newDiscoveryService(DefaultDiscoveryServiceProvider.java:29) at com.hazelcast.instance.impl.Node.createDiscoveryService(Node.java:391) at com.hazelcast.instance.impl.Node.(Node.java:294) at com.hazelcast.instance.impl.HazelcastInstanceImpl.createNode(HazelcastInstanceImpl.java:150) at com.hazelcast.instance.impl.HazelcastInstanceImpl.(HazelcastInstanceImpl.java:118) at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:218) at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:197) at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:135) at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61) at com.config.HazelcastConfig.embeddedHazelcast(HazelcastConfig.java:149) at com.config.HazelcastConfig$$SpringCGLIB$$0.CGLIB$embeddedHazelcast$1() at comconfig.HazelcastConfig$$SpringCGLIB$$FastClass$$1.invoke() at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:348) at com.comptel.catalog.config.HazelcastConfig$$SpringCGLIB$$0.embeddedHazelcast() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:146) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1355) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:785) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:768) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:145) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:509) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1439) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:599) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:785) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:768) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:145) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:509) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1439) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:599) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:380) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:351) at com.comptel.catalog.config.WebServiceConfig$$SpringCGLIB$$0.internalApiSessionInterceptor() at com.comptel.catalog.config.WebServiceConfig.addInterceptors(WebServiceConfig.java:111) at org.springframework.ws.config.annotation.WsConfigurerComposite.addInterceptors(WsConfigurerComposite.java:29) at org.springframework.ws.config.annotation.DelegatingWsConfiguration.addInterceptors(DelegatingWsConfiguration.java:33) at org.springframework.ws.config.annotation.WsConfigurationSupport.getInterceptors(WsConfigurationSupport.java:122) at org.springframework.ws.config.annotation.WsConfigurationSupport.annotationActionEndpointMapping(WsConfigurationSupport.java:111) at org.springframework.ws.config.annotation.DelegatingWsConfiguration$$SpringCGLIB$$0.CGLIB$annotationActionEndpointMapping$4() at org.springframework.ws.config.annotation.DelegatingWsConfiguration$$SpringCGLIB$$FastClass$$0.invoke() at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:348) at org.springframework.ws.config.annotation.DelegatingWsConfiguration$$SpringCGLIB$$0.annotationActionEndpointMapping() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:146) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1355) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205) at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:265) at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:806) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:609) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:174) at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:154) at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:96) at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:171) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4414) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:599) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:571) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:654) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:969) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1911) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:771) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:423) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1629) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:303) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:114) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:402) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:345) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:776) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:772) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1203) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1193) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:749) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:203) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:415) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:874) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171) at org.apache.catalina.startup.Catalina.start(Catalina.java:757) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473) Caused by: java.lang.ClassNotFoundException: java.net.http.HttpClient at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1332) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1144) ... 151 common frames omitted

and here is my HazelcastConfig.java class code snapshot

 @Bean
    public HazelcastInstance embeddedHazelcast() {
        Config hazelcastConfig = new Config();
        hazelcastConfig.setInstanceName("catalog");
        hazelcastConfig.setClusterName(clusterName);
        hazelcastConfig.getNetworkConfig().setPort(port).setPortAutoIncrement(false);
        hazelcastConfig.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
        hazelcastConfig.getNetworkConfig().getInterfaces().addInterface(networkInterface).setEnabled(false);
        if (StringUtils.isBlank(members)) {
            members = discover();

        } else {
            // add other hazelcast members including this instance
            members = networkInterface + ":" + hazelcastConfig.getNetworkConfig().getPort()
                    + (StringUtils.isNotBlank(members) ? "," : "") + members;
        }
        logger.info("Cache members List" + members);

        if(catCacheEnabled){
            if(isContainerisedEnv){
                hazelcastConfig.getNetworkConfig().getJoin().getKubernetesConfig().setEnabled(catCacheEnabled)
                        .setProperty("namespace",k8sNameSpace)
                        .setProperty("service-name",k8sServiceName);
            }else{
                hazelcastConfig.getNetworkConfig().getJoin().getTcpIpConfig().addMember(members).setEnabled(catCacheEnabled);
            }
        }else{
            hazelcastConfig.getNetworkConfig().getJoin().getAutoDetectionConfig().setEnabled(catCacheEnabled);  //Cache is disabled
        }

        HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(hazelcastConfig);
        return hazelcastInstance;
    }

With Springboot3.3.3 which is using Spring Frameowkr 6.1 any changes for Kubernates autoconfiguration ?

I am using Java17 Rocky9 Tomcat:10.1.26-1.1-rocky9-jre17

Same code base working fine if i will move again Springboot3.2.2

Answer 1

I am able to fix the issue, as per my analysis i was getting the issue with Hazel-cast member formation in Cloud Environment.

hazelcastConfig.getNetworkConfig().getJoin().getKubernetesConfig().setEnabled(catCacheEnabled)
                        .setProperty("namespace",k8sNameSpace)
                        .setProperty("service-name",k8sServiceName);

So upto Springboot3.2.2 this code was working fine and its started failing from Springboot3.3.3 and even Springboot3.3.4.

So with Springboot and upgraded to Tomcat version as well from 10.1.24 to 10.1.26 But Springboot3.2.2 was working fine both the version of Tomcat10.1.24 as well as 10.1.26 But Springboot3.3.3 failing with Tomcat so its evident that some where issue with Springboot version upgrade only.

So when Hazelcast trying to add members from Kubernetes environment ,some where internally its looking for java/net/http/HttpClient class and its not able to find it.Even i added dependency in pom.xml for this class but i was getting same issue.

So it look like
Spring Boot 3.3.x defaults to using the JDK's HttpClient for various internal and external HTTP requests and its possible this module in not present in the installed JDK in Cloud environment. So i run the command(java --list-modules) in pod to check what all modules available When i run same command in local machine it was showing more modules as compare to POD.

So this was the point from where i was confirmed issues with the Springboot version upgrade only creating this issue and product not able to deploy successfully.

May be in Spring Boot 3.2.2, this dependency was likely handled differently while in latest Springboot release ,things optimized things and using maximum modules of JDK in place of using from project library folder so Spring should align with JDK features and beyond. So fix this issue we have to make sure we are installing Full JDK with all the modules in place of JDK-min or JDK-micro image in container installation

Answer 2

Looks like you should upgrade the hazelcast-spring artifact and make sure you are indeed running jre17.