CODEWITHSUNDEEP
javascalaapache-storm
ethrbunny
ethrbunny

Reputation: 10469

Kafka + Storm - satisfying dependencies

I'm attempting to deploy my first topology to a storm cluster as part of an assessment for my company. The topology is just to get values from kafka and put them into cassandra and redis.

After copying over scads of .jar files to try to satisfy the various dependencies I've run into a issue where storm claims a dependency is missing but the startup class list in the logs shows the class as available.

Here's the exception:

java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
at kafka.utils.Pool.(Pool.scala:28) ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.consumer.FetchRequestAndResponseStatsRegistry$. ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.consumer.FetchRequestAndResponseStatsRegistry$.(FetchRequestAndResponseStats.scala) ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.consumer.SimpleConsumer.(SimpleConsumer.scala:39) ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.javaapi.consumer.SimpleConsumer.(SimpleConsumer.scala:34) ~[kafka_2.10-0.8.1.1.jar:na]
at storm.kafka.DynamicPartitionConnections.register(DynamicPartitionConnections.java:60) ~[storm-kafka-0.9.4.jar:0.9.4]
at storm.kafka.PartitionManager.(PartitionManager.java:64) ~[storm-kafka-0.9.4.jar:0.9.4]
at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) ~[storm-kafka-0.9.4.jar:0.9.4]
at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) ~[storm-kafka-0.9.4.jar:0.9.4]
at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135) ~[storm-kafka-0.9.4.jar:0.9.4]
at backtype.storm.daemon.executor$fn__4654$fn__4669$fn__4698.invoke(executor.clj:565) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463) ~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

Caused by: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_45]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_45]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[na:1.8.0_45]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_45]

When I look at the startup info for the supervisor thread I see this:

2015-06-07T07:55:19.941-0700 o.a.z.ZooKeeper [INFO] Client environment:java.class.path= ... /usr/local/src/apache-storm-0.9.4/lib/scala-library-2.11.6.jar: ...

When I open this file I see this entry:

-rwxrwxrwx 0 0 0 0 Mar 18 2014 scala/collection/GenTraversableOnce.class

So something else is amiss. What step(s) have I missed here?

NOTE: similar issues from org/jboss/netty/channel/ChannelFactory..

Upvotes: 1

Views: 1317

Answers (2)

user3072851
user3072851

Reputation: 41

Release Notes Source download: kafka-0.11.0.1-src.tgz (asc, md5)

Binary downloads: Scala 2.11  - kafka_2.11-0.11.0.1.tgz (asc, md5)
Scala 2.12  - kafka_2.12-0.11.0.1.tgz (asc, md5) We build for multiple

This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version should work (2.11 is recommended).

Upvotes: 0

Jerome Anthony
Jerome Anthony

Reputation: 8021

The Kafka version specifies which Scala version it is built against.

Scala 2.10 - kafka_2.10-0.9.0.1.tgz (asc, md5)

Scala 2.11 - kafka_2.11-0.9.0.1.tgz (asc, md5)

I made the mistake of using Scala 2.10 with kafka 2.11. I was able to resolve this by correcting my maven dependancies to the correct scala and kafka combination.

Upvotes: 0

Related Questions