CODEWITHSUNDEEP
scalaapache-sparksbt
k_trader
k_trader

Reputation: 53

Check a code of Spark RDD using local REPL

I want to check my scala spark code by REPL.
This code works well in spark-shell, but local scala with sbt didn't work.
(After executing sbt in the command line, and execute console command to start REPL.)

What should I do?

code

import org.apache.spark.SparkContext  
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

val conf = new SparkConf().setAppName("RddSample").setMaster("local[*]")
val sc = new SparkContext(conf)
val data = (0 to 10).toArray
val inputRDD = sc.parallelize(data)

error 

com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.9.4 requires Jackson Databind version >= 2.9.0 and < 2.10.0 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
  at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:18)
  at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:692)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:821)
... 40 elided

build.sbt

name := "Sratup"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "2.3.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % sparkVersion
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.4"
libraryDependencies +="com.fasterxml.jackson.core" % "jackson-databind" % "2.9.4"
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.4"
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.4"

Upvotes: 1

Views: 176

Answers (1)

SCouto
SCouto

Reputation: 7928

That Spark version, (in fact, AFAIK, every version up from 2.1.0 )already includes com.fasterxml.jackson.core so you don't need to put them in your build.sbt.

If you want to use a different version of them you should remove them from libraryDependencies and include them in dependencyOverrides

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala" % "2.9.4"

You can check which dependencies the spark 2.3.0 includes here

Upvotes: 0

Related Questions