EmbeddedKafka tests failing with java.lang.NoSuchFieldError: id

Question

When upgrading a component from Spring Boot 2.7.18 to Spring Boot 3.2.2, we are seeing all spring-kafka EmbeddedKafka tests fail with the following Spring initializationError:

java.lang.NoSuchFieldError: id

                at kafka.server.Defaults$.<clinit>(KafkaConfig.scala:187) ~[kafka_2.13-3.6.1.jar:na]

                at kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala:746) ~[kafka_2.13-3.6.1.jar:na]

                at kafka.server.KafkaConfig.DeleteTopicEnableProp(KafkaConfig.scala) ~[kafka_2.13-3.6.1.jar:na]

                at org.springframework.kafka.test.EmbeddedKafkaKraftBroker.addDefaultBrokerPropsIfAbsent(EmbeddedKafkaKraftBroker.java:256) ~[spring-kafka-test-3.1.1.jar:3.1.1]

                at org.springframework.kafka.test.EmbeddedKafkaKraftBroker.afterPropertiesSet(EmbeddedKafkaKraftBroker.java:202) ~[spring-kafka-test-3.1.1.jar:3.1.1]

                at org.springframework.kafka.test.context.EmbeddedKafkaContextCustomizer.customizeContext(EmbeddedKafkaContextCustomizer.java:130) ~[spring-kafka-test-3.1.1.jar:3.1.1]

                at org.springframework.boot.test.context.SpringBootContextLoader$ContextCustomizerAdapter.initialize(SpringBootContextLoader.java:442) ~[spring-boot-test-3.2.2.jar:3.2.2]

                at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:627) ~[spring-boot-3.2.2.jar:3.2.2]

                at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:400) ~[spring-boot-3.2.2.jar:3.2.2]

                at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) ~[spring-boot-3.2.2.jar:3.2.2]

                at org.springframework.boot.test.context.SpringBootContextLoader.lambda$loadContext$3(SpringBootContextLoader.java:137) ~[spring-boot-test-3.2.2.jar:3.2.2]

                at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:58) ~[spring-core-6.1.3.jar:6.1.3]

                at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:46) ~[spring-core-6.1.3.jar:6.1.3]

                at org.springframework.boot.SpringApplication.withHook(SpringApplication.java:1454) ~[spring-boot-3.2.2.jar:3.2.2]

                at org.springframework.boot.test.context.SpringBootContextLoader$ContextLoaderHook.run(SpringBootContextLoader.java:552) ~[spring-boot-test-3.2.2.jar:3.2.2]

Has anyone seen this or have any suggestions for how to resolve it?

From the stack trace, we see it cannot find an "id" field.

But not sure what this "id" is.

How can we troubleshoot spring-kafka code which is invoking scala code? The line numbers in Intellij console don't seem to map to the actual scala code.

The original versions used with Spring Boot 2.7.18: spring-kafka 2.9.13, kafka-clients 3.2.3

The versions used with Spring Boot 3.2.2: spring-kafka 3.1.1, kafka-clients 3.6.1

Here is a list of the dependencies/versions being used with Spring Boot 3.2.2.

dependencies {

    api platform (group: 'org.springframework.boot', name: 'spring-boot-dependencies', version: "3.2.2")

 

    api (group: 'org.springframework.boot', name: 'spring-boot-configuration-processor')

    api (group: 'org.springframework.boot', name: 'spring-boot-starter-actuator')

    api (group: 'org.springframework.boot', name: 'spring-boot-starter-data-jdbc')

    api (group: 'org.springframework.boot', name: 'spring-boot-starter-json')

    api (group: 'org.springframework.boot', name: 'spring-boot-starter-web')

    api (group: 'org.springframework.boot', name: 'spring-boot-starter-security')

    api (group: 'org.springframework.kafka', name: 'spring-kafka')

 

    api (group: 'com.google.guava', name: 'guava', version: '33.0.0-jre')

    api (group: 'com.sapiens.bdms', name: 'bdms-java-exe-helper', version: '7.1')

    api (group: 'jakarta.validation', name: 'jakarta.validation-api', version: '3.0.2')

    api (group: 'org.ehcache', name: 'ehcache', version: '3.10.8')

    api (group: 'org.apache.avro', name: 'avro', version: '1.11.3')

    api (group: 'org.apache.httpcomponents.client5', name: 'httpclient5', version: '5.3')

    api (group: 'org.hibernate.validator', name: 'hibernate-validator', version: '8.0.1.Final')

 

    api (group: 'redis.clients', name: 'jedis', version: '5.1.0')

 

    implementation (group: 'org.apache.tika', name: 'tika-core', version: '2.9.1')

 

    // lombok

    implementation (group: 'org.projectlombok', name: 'lombok', version: "1.18.30")

    annotationProcessor (group: 'org.projectlombok', name: 'lombok', version: "1.18.30")

    testImplementation (group: 'org.projectlombok', name: 'lombok', version: "1.18.30")

    testAnnotationProcessor (group: 'org.projectlombok', name: 'lombok', version: "1.18.30")

 

    testImplementation (group: 'com.h2database', name: 'h2')

    testImplementation (group: 'org.wiremock', name: 'wiremock', version: '3.3.1'){

        exclude group: 'org.slf4j', module: 'slf4j-api'

    }

    testImplementation (group: 'io.confluent', name: 'kafka-avro-serializer', version: '7.5.1')

    testImplementation (group: 'io.micrometer', name: 'micrometer-registry-prometheus', version: '1.12.1')

    testImplementation (group: 'it.ozimov', name: 'embedded-redis', version: '0.7.3'){

        exclude group: 'org.slf4j', module: 'slf4j-simple'

    }

    testImplementation (group: 'org.awaitility', name: 'awaitility', version: '4.2.0')

    testImplementation (group: 'jakarta.el', name: 'jakarta.el-api', version: '5.0.1')

    testImplementation (group: 'org.springframework.boot', name: 'spring-boot-starter-test')

    testImplementation (group: 'org.springframework.kafka', name: 'spring-kafka-test')

    testImplementation (group: 'org.springframework.boot', name: 'spring-boot-starter-aop')

    testImplementation (group: 'junit', name: 'junit', version: '4.13.2')

 

    testImplementation('org.scala-lang:scala3-library_3:3.3.1')

}

 

We have other components on Spring Boot 3.2.2 that are successfully running EmbeddedKafka tests.

I have compared dependency trees, and have yet to find a difference that would cause the NoSuchFieldError.

Answer 1

This is due to a dependency misconfiguration. In other words, whatever the main dependency (e.g., spring-kafka) depends on as far as kafka-clients is concerned does not match.

I hit this issue while setting up embedded-kafka-schema-registry dependency (and is not exactly what you've been doing) but the explanation is the same.

How could I explain this error: java.lang.NoSuchFieldError: id, you guess?

Looking at the stack trace shows the side effect of this dependency mismatch.

The line kafka.server.Defaults$.<clinit>(KafkaConfig.scala:187) ~[kafka_2.13-3.6.1.jar:na] is this line in the code:

val OffsetsTopicCompressionCodec: Int = OffsetConfig.DefaultOffsetsTopicCompressionType.id

Can you see this id? That's what's missing in whatever OffsetConfig.DefaultOffsetsTopicCompressionType points to (that comes from the mismatched version).

This is all within kafka main dependency (where scala/kafka/server/KafkaConfig.scala lives). That's one of the dependencies (perhaps even among the transitive ones = not explicitly specified by you).

The other dependency is the one with CompressionType class. This is kafka-clients.

kafka-clients has to be API-compatible with the core Kafka library (kafka) so they can work together or...you run into this nasty java.lang.NoSuchFieldError: id error.