CODEWITHSUNDEEP
scalaapache-sparkapache-kafkaspark-streaming
Gnana
Gnana

Reputation: 2230

Spark Kafka Stream error

I am new Spark and Kafka. I have done Spark and Kafka setup done in Windows system. Both are working good. I have followed below mentioned tutorial and executing the Scala code in Spark-shell and getting below mentioned error.

Can anybody help me on how to listen Kafka stream using Spark in Scala.

Spark : 2.2 Kafka : 2.12

http://www.godatafy.com/poc/streaming-with-spark-kafka/

spark-shell -jars ..\jars\spark-streaming-kafka_2.11-1.6.3.jar

scala> import org.apache.spark.SparkConf
import org.apache.spark.SparkConf

scala> import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.StreamingContext

scala> import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.Seconds

scala> import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.kafka.KafkaUtils

scala> sc.stop

scala> val sparkConf = new SparkConf().setAppName("KafkaWordCount").setMaster("local[2]")
sparkConf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@427c2c96

scala> val ssc = new StreamingContext(sparkConf, Seconds(2))
ssc: org.apache.spark.streaming.StreamingContext = org.apache.spark.streaming.StreamingContext@1fd73dcb

scala> val lines = KafkaUtils.createStream(ssc, "localhost:2181", "spark-streaming-consumer-group", Map("test" -> 5))
error: missing or invalid dependency detected while loading class file 'KafkaUtils.class'.
Could not access term kafka in package <root>,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of <root>.

Upvotes: 0

Views: 1529

Answers (1)

koiralo
koiralo

Reputation: 23119

If you are using Kafka 0.10 or higher you need to add the jar to the version that supports spark 2.2

Use spark-streaming-kafka-0-10-2.10-2.2.0.jar

rather than spark-streaming-kafka_2.11-1.6.3.jar

I hope this solves your problem!

Upvotes: 1

Related Questions