KafkaEmbedded throwing ClassNotFoundException for SecurityProtocol

Question

My production code is working fine, but my integration test is throwing ClassNotFoundException for SecurityProtocol. I am using spring-boot-parent 1.5.10.RELEASE.

    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
        <version>1.3.3.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <version>1.3.3.RELEASE</version>
        <scope>test</scope>
    </dependency>

I am getting the below exception:

java.lang.NoClassDefFoundError: org/apache/kafka/common/security/auth/SecurityProtocol

    at kafka.server.KafkaConfig$.<init>(KafkaConfig.scala:578)
    at kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala)
    at kafka.utils.TestUtils$.createBrokerConfig(TestUtils.scala:228)
    at kafka.utils.TestUtils.createBrokerConfig(TestUtils.scala)
    at org.springframework.kafka.test.rule.KafkaEmbedded.createProperties(KafkaEmbedded.java:214)
    at org.springframework.kafka.test.rule.KafkaEmbedded.before(KafkaEmbedded.java:191)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.security.auth.SecurityProtocol
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 15 more

When I am using this pom, I am getting a different error:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <scope>test</scope>
</dependency>

java.lang.NoClassDefFoundError: kafka/utils/Time
    at com.betstars.eventdata.app.kafka.message.KafkaMessageSenderTest.<clinit>(KafkaMessageSenderTest.java:39)
    at sun.misc.Unsafe.ensureClassInitialized(Native Method)
    at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
    at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
    at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
    at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
    at java.lang.reflect.Field.get(Field.java:393)
    at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
    at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
    at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
    at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
    at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.ClassNotFoundException: kafka.utils.Time
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 19 more

And when I use the latest version, I am getting a different error:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.1.2.RELEASE</version>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <version>2.1.2.RELEASE</version>
    <scope>test</scope>
</dependency>

java.lang.NoSuchMethodError: org.springframework.util.Assert.state(ZLjava/util/function/Supplier;)V

    at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.determineInferredType(MessagingMessageListenerAdapter.java:396)
    at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.<init>(MessagingMessageListenerAdapter.java:100)
    at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.<init>(RecordMessagingMessageListenerAdapter.java:61)
    at org.springframework.kafka.config.MethodKafkaListenerEndpoint.createMessageListenerInstance(MethodKafkaListenerEndpoint.java:172)
    at org.springframework.kafka.config.MethodKafkaListenerEndpoint.createMessageListener(MethodKafkaListenerEndpoint.java:132)
    at org.springframework.kafka.config.AbstractKafkaListenerEndpoint.setupMessageListener(AbstractKafkaListenerEndpoint.java:355)
    at org.springframework.kafka.config.AbstractKafkaListenerEndpoint.setupListenerContainer(AbstractKafkaListenerEndpoint.java:340)

Answer 1

As a complement to the accepted answer I will described my solution because in my case the class not found was protocol/SecurityProtocol

java.lang.NoClassDefFoundError: org/apache/kafka/common/protocol/SecurityProtocol

which is different of the security/auth/SecurityProtocol

java.lang.NoClassDefFoundError: org/apache/kafka/common/security/auth/SecurityProtocol

I basically had to check all my dependencies using ./mvnw dependency:tree and exclude kafka-clients that was not on the version that my project was.

<exclusions>
  <exclusion>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.11</artifactId>
  </exclusion>
  <exclusion>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
  </exclusion>
</exclusions>

most of them was version 3.0. At the Apache Kafka repo there is a hint showing that it has to be compatible with the scala version. Then it was also necessary to verify all the scala version on the project. I had a few that were using different version.

Scala 2.12 support has been deprecated since Apache Kafka 3.0

and add the only version that has the org/apache/kafka/common/protocol/SecurityProtocol class and it is compatible with scala 2.11.

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>0.11.0.0</version>
</dependency>

Answer 2

java.lang.NoClassDefFoundError: org/apache/kafka/common/security/auth/SecurityProtocol

This is caused by a mismatched kafka_2.11 and kafka-clients jars; they must be the same version. SecurityProtocol was moved to a different package in the 1.0.0 kafka_clients jar.

java.lang.NoSuchMethodError: org.springframework.util.Assert.state(ZLjava/util/function/Supplier;)V

This one is because spring-kafka 2.x requires Spring Framework 5.0 which is not compatible with boot 1.5; it uses Spring Framework 4.3.x.

Spring Boot 2.0 will be released soon, which will work with the latest Spring Kafka.

java.lang.NoClassDefFoundError: kafka/utils/Time

That's another jar mismatch

KAFKA-2247; Merge kafka.utils.Time and kafka.common.utils.Time

Because of the way boot does dependency management, when you override the versions of, say, spring-kafka - you also have to override the dependencies to the right versions too (kafka-clients, kafka_2.11).

Answer 3

Surprisingly my server was running, and so my integration test was failing. I stopped my running server and started the test, and it worked fine, without any explicit version definition.