CODEWITHSUNDEEP
jsonscalajackson
J.Done
J.Done

Reputation: 3053

Jackson library version confilct

dependencies {
    compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.1'
    compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.1.0'
    compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.1.0'
    compile group: 'org.elasticsearch', name: 'elasticsearch-spark-20_2.11', version: '5.1.2'

It's a part of my gradle configuration. and It works find. When I added 'play-json' library like below to process json string.

compile group: 'com.typesafe.play', name: 'play-json_2.11', version: '2.5.13'

Errors comes out where scala try to read csv file.

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.8 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)

    spark.read
        .option("header", fileHeader)
        .option("charset", charset)
        .csv( "./data/" + filePath) // error here

I'm working with scala 2.11 and spark 2.1.0. Any Idea of this?

Upvotes: 0

Views: 1702

Answers (3)

ForeverLearner
ForeverLearner

Reputation: 2113

If you are using the latest spark version 3.0.0-preview2, the below configuration is a working build.sbt:

name := "coursera"

version := "0.1"

scalaVersion := "2.12.10"
val sparkVersion = "3.0.0-preview2"
val playVersion="2.8.1"

val jacksonVersion="2.10.1"

//override if you wish to
//dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % jacksonVersion
//dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "com.typesafe.play" %% "play-json" % playVersion
)

Upvotes: 0

Mikel San Vicente
Mikel San Vicente

Reputation: 3863

Play 2.4.10 works well with Spark 2.1 because the jackson they use are similar

dependencies {
    compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.1'
    compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.1.0'
    compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.1.0'
    compile group: 'org.elasticsearch', name: 'elasticsearch-spark-20_2.11', version: '5.1.2'
    compile group: 'com.typesafe.play', name: 'play-json_2.11', version: '2.4.10'

Upvotes: 2

Vidya
Vidya

Reputation: 30310

The problem is that Spark has a dependency on an older version of Jackson and the Jackson Scala module. I imagine the module isn't playing nice with the more recent version of Jackson that comes along as a dependency of play-json.

Try excluding Jackson as a transitive dependency from Spark like this:

compile('org.apache.spark:spark-core_2.11:2.1.0') {
     exclude group: 'com.fasterxml.jackson.core', module: 'jackson-databind' 
     exclude group: 'com.fasterxml.jackson.module', module: 'jackson-module-scala_2.11' 
}

You could also try adding a direct dependency on Jackson 2.7.8 as well.

Upvotes: 2

Related Questions