spring-kafka 2.0.x kafka .11 and spring-boot 2.0.0.RELEASE compatibility issues: java.lang.NoSuchMethodError

Question

When upgrading from spring-boot 2.0.0.Mx to 2.0.0.RELEASE it looks like there are incompatibilities between spring-boot, spring-kafka 2.0.x, and kafka .11. The compatibility issues are clearly defined between spring-kafka and the kafka version here. However when using the compatible versions mentioned and upgrading to spring-boot 2.0.0.RELEASE the following exception is thrown.

        Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
    2018-04-12 10:48:34.501 ERROR 13141 --- [           main] o.s.boot.SpringApplication               : Application run failed

    org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaListenerContainerFactory' defined in class path resource [org/springframework/boot/autoconfigure/kafka/KafkaAnnotationDrivenConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory]: Factory method 'kafkaListenerContainerFactory' threw exception; nested exception is java.lang.BootstrapMethodError: java.lang.NoSuchMethodError: org.springframework.kafka.listener.config.ContainerProperties.setClientId(Ljava/lang/String;)V
            at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
            at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
            at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
            at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
            at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
            at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:758)
            at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868)
            at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
            at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
            at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:752)
            at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:388)
            at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
            at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246)
            at org.springframework.boot.SpringApplication.run(SpringApplication.java:1234)
            at Server.main(Server.kt:19)
    Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory]: Factory method 'kafkaListenerContainerFactory' threw exception; nested exception is java.lang.BootstrapMethodError: java.lang.NoSuchMethodError: org.springframework.kafka.listener.config.ContainerProperties.setClientId(Ljava/lang/String;)V
            at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
            at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
            ... 18 common frames omitted
    Caused by: java.lang.BootstrapMethodError: java.lang.NoSuchMethodError: org.springframework.kafka.listener.config.ContainerProperties.setClientId(Ljava/lang/String;)V
            at org.springframework.boot.autoconfigure.kafka.ConcurrentKafkaListenerContainerFactoryConfigurer.configureContainer(ConcurrentKafkaListenerContainerFactoryConfigurer.java:99)
            at org.springframework.boot.autoconfigure.kafka.ConcurrentKafkaListenerContainerFactoryConfigurer.configure(ConcurrentKafkaListenerContainerFactoryConfigurer.java:80)
            at org.springframework.boot.autoconfigure.kafka.KafkaAnnotationDrivenConfiguration.kafkaListenerContainerFactory(KafkaAnnotationDrivenConfiguration.java:72)
            at org.springframework.boot.autoconfigure.kafka.KafkaAnnotationDrivenConfiguration$$EnhancerBySpringCGLIB$$b9d721e8.CGLIB$kafkaListenerContainerFactory$1(<generated>)
            at org.springframework.boot.autoconfigure.kafka.KafkaAnnotationDrivenConfiguration$$EnhancerBySpringCGLIB$$b9d721e8$$FastClassBySpringCGLIB$$25044a84.invoke(<generated>)
            at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
            at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
            at org.springframework.boot.autoconfigure.kafka.KafkaAnnotationDrivenConfiguration$$EnhancerBySpringCGLIB$$b9d721e8.kafkaListenerContainerFactory(<generated>)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
            ... 19 common frames omitted
    Caused by: java.lang.NoSuchMethodError: org.springframework.kafka.listener.config.ContainerProperties.setClientId(Ljava/lang/String;)V
            at java.lang.invoke.MethodHandleNatives.resolve(Native Method)
            at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:975)
            at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1000)
            at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1394)
            at java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1750)
            at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:477)
            ... 32 common frames omitted

According to Artem Bilan in this random gitter thread, spring-boot 2 is only compatible with spring-kafka 2.1.x. I couldn't find any documentation stating this.

Given all of this these are my only options:

1. Upgrade to kafka 1.0 because spring-kafka 2.1 is only compatible with kafka 1.0.

2. Stick with an older version of spring-boot.

3. Configure kafka manually (I haven't proven this will work but would make sense).

Is there something in the works to allow compatibility between spring-boot 2 and older version of spring-kafka?

Answer 1

I do like this in POM to downgrade Spring Kafka:

<properties>
    <spring-kafka.version>2.0.4.RELEASE</spring-kafka.version>
</properties>

Then I get the same NoSuchMethodError. From there I jumped to the ConcurrentKafkaListenerContainerFactoryConfigurer and decided to override it according KafkaAnnotationDrivenConfiguration:

@Bean
public ConcurrentKafkaListenerContainerFactoryConfigurer kafkaListenerContainerFactoryConfigurer(
        KafkaProperties kafkaProperties,
        ObjectProvider<RecordMessageConverter> messageConverterObjectProvider,
        ObjectProvider<KafkaTemplate<Object, Object>> kafkaTemplateObjectProvider) {

    RecordMessageConverter messageConverter = messageConverterObjectProvider.getIfUnique();
    KafkaTemplate<Object, Object> kafkaTemplate = kafkaTemplateObjectProvider.getIfUnique();

    return new ConcurrentKafkaListenerContainerFactoryConfigurer() {

        @Override
        public void configure(ConcurrentKafkaListenerContainerFactory<Object, Object> listenerFactory,
                ConsumerFactory<Object, Object> consumerFactory) {

            listenerFactory.setConsumerFactory(consumerFactory);
            configureListenerFactory(listenerFactory);
            configureContainer(listenerFactory.getContainerProperties());
        }

        private void configureListenerFactory(
                ConcurrentKafkaListenerContainerFactory<Object, Object> factory) {
            PropertyMapper map = PropertyMapper.get();
            KafkaProperties.Listener properties = kafkaProperties.getListener();
            map.from(properties::getConcurrency).whenNonNull().to(factory::setConcurrency);
            map.from(() -> messageConverter).whenNonNull()
                    .to(factory::setMessageConverter);
            map.from(() -> kafkaTemplate).whenNonNull().to(factory::setReplyTemplate);
            map.from(properties::getType).whenEqualTo(KafkaProperties.Listener.Type.BATCH)
                    .toCall(() -> factory.setBatchListener(true));
        }

        private void configureContainer(ContainerProperties container) {
            PropertyMapper map = PropertyMapper.get();
            KafkaProperties.Listener properties = kafkaProperties.getListener();
            map.from(properties::getAckMode).whenNonNull().to(container::setAckMode);
            map.from(properties::getAckCount).whenNonNull().to(container::setAckCount);
            map.from(properties::getAckTime).whenNonNull().as(Duration::toMillis)
                    .to(container::setAckTime);
            map.from(properties::getPollTimeout).whenNonNull().as(Duration::toMillis)
                    .to(container::setPollTimeout);
            map.from(properties::getNoPollThreshold).whenNonNull()
                    .to(container::setNoPollThreshold);
            map.from(properties::getIdleEventInterval).whenNonNull().as(Duration::toMillis)
                    .to(container::setIdleEventInterval);
            map.from(properties::getMonitorInterval).whenNonNull().as(Duration::getSeconds)
                    .as(Number::intValue).to(container::setMonitorInterval);
        }

    };
}

Right, that's mostly copy-paste, but at least I stop to get that exception and propagate as much properties as possible.