CODEWITHSUNDEEP
scalaapache-sparksbt
David Portabella
David Portabella

Reputation: 12720

override guava dependency version of spark

spark depends on an old version of guava. i build my spark project with sbt assembly, excluding spark using provided, and including the latest version of guava.

However, when running sbt-assembly, the guava dependency is excluded also from the jar.

my build.sbt:

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
  "com.google.guava" % "guava" % "11.0"
)

if i remove the % "provided", then both spark and guava is included.

so, how can i exclude spark and include guava?

Upvotes: 3

Views: 3008

Answers (1)

Assaf Mendelson
Assaf Mendelson

Reputation: 13001

You are looking for shading options. See here but basically you need to add shading instructions. Something like this:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.guava.**" -> "my_conf.@1")
    .inLibrary("com.google.guava" % "config" % "11.0")
    .inProject
  )

There is also the corresponding maven-shade-plugin for those who prefer maven.

Upvotes: 2

Related Questions