CODEWITHSUNDEEP
javaapache-sparksbtsbt-assembly
Georg Heiler
Georg Heiler

Reputation: 17724

sbt assembly switch between provided dependencies

I am developing a spark application. To test it locally I want to run sbt run. This requires the dependencies to be available locally. But I also want to use sbt assembly to generate a jar which can be used for spark-submit that jar should only include some of the dependencies e.g. not include the spark dependencies.

When I mark a dependency in build.sbt as % "provided" it no longer is available via sbt run

Upvotes: 2

Views: 615

Answers (1)

FaigB
FaigB

Reputation: 2281

If the case to exclude jars in assembly use in main built.sbt

excludedJars in assembly := {
  val cp = (fullClasspath in assembly).value
  cp filter {f =>
     f.data.getName.contains("spark"),
     f.data.getName.startsWith("jar_name")
  }
}

Upvotes: 2

Related Questions