Storm1.1.0 + kafka0.10.2.1: NoSuchMethodError&ExceptionInInitializerError

Question

I am new to storm and kafka. I want to run a open source project:github:cestella:streaming_outliers

This project uses storm0.10.0. However, My storm cluster is storm1.0.3, which is not work for storm0.10.0.

So,I have to change the version of storm and some related packages.

Then, when I run this topology, it will turn out the following errors.

java.lang.NoSuchMethodError: org.apache.kafka.common.network.NetworkSend.(Ljava/lang/String;Ljava/nio/ByteBuffer;)V at kafka.network.RequestOrResponseSend.(RequestOrResponseSend.scala:41) at kafka.network.RequestOrResponseSend.(RequestOrResponseSend.scala:44) at kafka.network.BlockingChannel.send(BlockingChannel.scala:112) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:98) at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83) at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:149) at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79) at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:75) at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:65) at org.apache.storm.kafka.PartitionManager.(PartitionManager.java:103) at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129) at org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:644) at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)

And

java.lang.ExceptionInInitializerError at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:90) at kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:26) at kafka.consumer.FetchRequestAndResponseMetrics.(FetchRequestAndResponseStats.scala:35) at kafka.consumer.FetchRequestAndResponseStats.(FetchRequestAndResponseStats.scala:47) at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:60) at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:60) at kafka.utils.Pool$$anonfun$getAndMaybePut$1.apply(Pool.scala:52) at kafka.utils.Pool.getAndMaybePut(Pool.scala:70) at kafka.utils.Pool.getAndMaybePut(Pool.scala:52) at kafka.consumer.FetchRequestAndResponseStatsRegistry$.getFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:64) at kafka.consumer.SimpleConsumer.(SimpleConsumer.scala:44) at kafka.javaapi.consumer.SimpleConsumer.(SimpleConsumer.scala:34) at org.apache.storm.kafka.DynamicPartitionConnections.register(DynamicPartitionConnections.java:60) at org.apache.storm.kafka.PartitionManager.(PartitionManager.java:74) at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129) at org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:644) at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) at java.lang.Runtime.addShutdownHook(Runtime.java:211) at com.yammer.metrics.Metrics.(Metrics.java:21) ... 21 more

And

java.lang.NullPointerException at net.opentsdb.utils.Config.getBoolean(Config.java:354) ~[stormjar.jar:1.6.6] at net.opentsdb.utils.Config.loadStaticVariables(Config.java:620) ~[stormjar.jar:1.6.6] at net.opentsdb.utils.Config.setDefaults(Config.java:532) ~[stormjar.jar:1.6.6] at net.opentsdb.utils.Config.(Config.java:131) ~[stormjar.jar:1.6.6] at com.caseystella.analytics.timeseries.tsdb.TSDBHandler$TSDBConfig.(TSDBHandler.java:127) ~[stormjar.jar:1.6.6] at com.caseystella.analytics.timeseries.tsdb.TSDBHandler$TSDBConfig.(TSDBHandler.java:124) ~[stormjar.jar:1.6.6] at com.caseystella.analytics.timeseries.tsdb.TSDBHandler.configure(TSDBHandler.java:144) ~[stormjar.jar:1.6.6] at com.caseystella.analytics.outlier.OutlierBolt.prepare(OutlierBolt.java:63) ~[stormjar.jar:1.6.6] at org.apache.storm.daemon.executor$fn__4973$fn__4986.invoke(executor.clj:791) ~[storm-core-1.0.3.jar:1.0.3] at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) [storm-core-1.0.3.jar:1.0.3] java.lang.NullPointerException at net.opentsdb.utils.Config.getBoolean(Config.java:354) ~[stormjar.jar:1.6.6] at net.opentsdb.utils.Config.loadStaticVariables(Config.java:620) ~[stormjar.jar:1.6.6] at net.opentsdb.utils.Config.setDefaults(Config.java:532) ~[stormjar.jar:1.6.6] at net.opentsdb.utils.Config.(Config.java:131) ~[stormjar.jar:1.6.6] at com.caseystella.analytics.timeseries.tsdb.TSDBHandler$TSDBConfig.(TSDBHandler.java:127) ~[stormjar.jar:1.6.6] at com.caseystella.analytics.timeseries.tsdb.TSDBHandler$TSDBConfig.(TSDBHandler.java:124) ~[stormjar.jar:1.6.6] at com.caseystella.analytics.timeseries.tsdb.TSDBHandler.configure(TSDBHandler.java:144) ~[stormjar.jar:1.6.6] at com.caseystella.analytics.outlier.OutlierBolt.prepare(OutlierBolt.java:63) ~[stormjar.jar:1.6.6] at org.apache.storm.daemon.executor$fn__4973$fn__4986.invoke(executor.clj:791) ~[storm-core-1.0.3.jar:1.0.3] at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) [storm-core-1.0.3.jar:1.0.3]

Following is the packages' version.

kafka_version: 0.10.2.1

kafka-clients version: 0.10.2.1

storm_version: 1.1.0

storm-kafka version: 1.1.0

flux_version: 1.1.0

tsdb_version: 2.3.0

There are some methods in the internet say that appending the dependency of kafka-clients can solve the first error. However,it don't work.

Thanks a lot.

Answer 1

kafka and kafka-clients version in pom.xml should match each other.

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.12</artifactId>
  <version>1.0.0</version>
</dependency>

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>1.0.0</version>
</dependency>

Answer 2

No such method error comes when your application contains multiple versions of the same jar (ex. kafka 0.10.2.1 & 0.10.0.1). Check your project dependencies and remove the older version of the jar.