CODEWITHSUNDEEP
scalaapache-sparksbtfasterxml
wongp
wongp

Reputation: 21

Scala module 2.8.11 requires Jackson Databind version >= 2.8.0 and < 2.9.0

I'm using Scala 2.11 and Spark 2.4.3 for our AWS glue jobs. Recently, I got the error message below in our the build pipeline.

Cause: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.8.11 requires Jackson Databind version >= 2.8.0 and < 2.9.0

I've tried -

  1. Change the the jackson-module-scala version from 2.8.11 to 2.12.0. This fixed the build pipeline, but I get a different error message in the glue job

Exception in User Class: java.lang.ExceptionInInitializerError.

  1. Change the jackson-module-scala version from 2.8.11 to 2.13.1. I refactored code to get the unit tests, and fix the build pipeline, but in the glue job get the error message below

Exception in User Class: java.lang.VerifyError : Bad return type

  1. I've tried added dependencyOverrides with jackson-module-scala version 2.12.0. The build pipeline would work, but the glue job would fail.
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"

Got any ideas what I'm doing wrong or how I can fix my issue?

See below for the build.sbt file

name in ThisBuild := "etl"
organization in ThisBuild := "XXXXXXXXXX"
scalaVersion in ThisBuild := "2.11.12"
version in ThisBuild := "0.2"

addCommandAlias("sanity", ";clean ;compile ;test ;scalafmtAll ;scalastyle ;assembly")

lazy val framework = project.settings(settings, libraryDependencies ++= commonDependencies)

lazy val scripts = project
  .settings(settings, libraryDependencies ++= commonDependencies)
  .dependsOn(framework % "compile->compile;test->test")

lazy val settings = Seq(
  test in assembly := {},
  scalacOptions ++= Seq(),
  resolvers ++= Seq(
    Resolver.sonatypeRepo("releases"),
    "aws-glue-etl-artifacts" at "https://aws-glue-etl-artifacts.s3.amazonaws.com/release/"
  ),
  assemblyMergeStrategy in assembly := {
    case PathList("META-INF", "io.netty.versions.properties", xs @ _*) => MergeStrategy.singleOrError
    case "module-info.class" => MergeStrategy.discard
    case x: String if x.contains("UnusedStubClass") => MergeStrategy.first
    case y =>
      val oldStrategy = (assemblyMergeStrategy in assembly).value
      oldStrategy(y)
  }
)

lazy val commonDependencies = Seq(
  "com.amazonaws" % "AWSGlueETL" % "1.0.0" % Provided,
  "com.databricks" %% "spark-xml" % "0.8.0",
  "org.scalatest" %% "scalatest" % "3.1.1" % Test,
  "org.scalamock" %% "scalamock" % "4.4.0" % Test,
  "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.8.11",
  "io.netty" % "netty-all" % "4.1.17.Final" % Test, 
  "org.apache.spark" %% "spark-avro" % "2.4.3",
  "com.crealytics" %% "spark-excel" % "0.13.6"
)


fork in ThisBuild := true
parallelExecution in ThisBuild := true
testForkedParallel in ThisBuild := false
logBuffered in ThisBuild := false
testOptions in ThisBuild += Tests.Argument(TestFrameworks.ScalaTest, "-oDFG")

javaOptions ++= Seq(
  "-XX:+CMSClassUnloadingEnabled",
  "-XX:MaxMetaspaceSize=512M",
  "-XX:MetaspaceSize=256M",
  "-Xms512M",
  "-Xmx2G",
  "-XX:MaxPermSize=2048M"
)

The function below uses the Jackson module scala 2.8.11 (up to 2.12.0)

import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper

object JsonUtils {
  private val mapper = new ObjectMapper() with ScalaObjectMapper
  mapper.registerModule(DefaultScalaModule)

  def fromJson[T](json: String)(implicit m: Manifest[T]): T = {
    mapper.readValue[T](json)
  }
}

Upvotes: 0

Views: 3478

Answers (2)

wongp
wongp

Reputation: 21

I've resolved my issue.

In the function, I changed the library import from

import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper

to

import com.fasterxml.jackson.module.scala.ScalaObjectMapper

In the build.sbt, I added the following lines

libraryDependencies += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.12.0"

dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.6.7.1"

Upvotes: 1

Hossein Torabi
Hossein Torabi

Reputation: 733

U need to override it, like this:

dependencyOverrides ++= Seq(
"com.fasterxml.jackson.core" % "jackson-databind" % versions("jackson"),
"com.fasterxml.jackson.core" % "jackson-core" % versions("jackson"))

Upvotes: 1

Related Questions