CODEWITHSUNDEEP
apache-sparkapache-kafkaspark-streamingjava-11
m b
m b

Reputation: 310

Spark Streaming Kafka with Java11 scala code Issue

I am using scala Spark Streaming Kafka with JDK 11. But I am getting the below error.

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

Below is the code I am using.

val conf = new SparkConf().setMaster("local[*]").setAppName("KafkaExample")
      .set("spark.mongodb.input.uri", "mongodb://127.0.0.1/db.table_data")
      .set("spark.mongodb.output.uri", "mongodb://127.0.0.1/db.table_data")
      .set("spark.driver.allowMultipleContexts", "false")
      .set("spark.ui.enabled", "false")

val kafkaParams = Map("metadata.broker.list" -> "localhost:9092")
    val topics = List("topic").toSet
    val lines = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
      ssc, kafkaParams, topics)

Below is my pom.xml

<dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-core_2.12</artifactId>
          <version>2.4.8</version><!--<version>2.3.2</version>-->
      </dependency>
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-sql_2.12</artifactId>
          <version>2.4.8</version>
      </dependency>
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-streaming_2.12</artifactId>
          <version>2.4.8</version><!--<version>2.3.2</version>-->
          <!--<scope>provided</scope>-->
      </dependency>
      <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-streaming-kafka_2.11</artifactId>
          <version>1.6.1</version>
      </dependency>
     
      <dependency>
          <groupId>org.mongodb.spark</groupId>
          <artifactId>mongo-spark-connector_2.12</artifactId>
          <version>2.4.3</version>
      </dependency>
      <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
      <dependency>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>jackson-core</artifactId>
          <version>2.10.0</version>
      </dependency>

The issue is coming in KafkaUtils.createDirectStream() and i checked the maven and scala compatibility JDK 11 is compatible with scala 2.12 and higher and we don't have any maven dependency for spark-streaming-kafka jar. KIndly let me know if my analysis is wrong and which jar should i use for spark-streaming-kafka for JDK 11

Upvotes: 0

Views: 464

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191884

  1. DirectStream API is deprecated, you should be using spark-sql-kafka-0-10 dependencies. The Mongo Spark driver works with Spark SQL also

  2. As commented, you're mixing your Scala versions (the broker Scala version doesn't matter), and you're also mixing your Spark versions

Upvotes: 0

Related Questions