CODEWITHSUNDEEP
scalaintellij-ideaapache-sparksbt
qed
qed

Reputation: 23134

Use spark in a sbt project in intellij

Here is the build.sbt file:

name := "scalaChartTest"

version := "1.0"

scalaVersion := "2.11.7"

//libraryDependencies += "org.jfree" % "jfreechart" % "1.0.19"
//
//libraryDependencies += "org.scalafx" %% "scalafx" % "8.0.0-R4"
//
//libraryDependencies += "com.github.wookietreiber" %% "scala-chart" % "latest.integration"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.4.1"

And I got an error after refreshing:

15:56:30 SBT project import
         [warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version:
         [warn]  * org.scala-lang:scala-compiler:(2.11.0, 2.11.7)
         [warn]  * org.scala-lang:scala-reflect:(2.11.2, 2.11.7)
         [warn]  * jline:jline:(0.9.94, 2.12.1)
         [warn]  * org.scala-lang.modules:scala-parser-combinators_2.11:(1.0.1, 1.0.4)
         [warn]  * org.scala-lang.modules:scala-xml_2.11:(1.0.1, 1.0.4)
         [warn]  * org.slf4j:slf4j-api:(1.6.4, 1.7.10)

What went wrong here?

Upvotes: 6

Views: 4326

Answers (2)

qed
qed

Reputation: 23134

I learned from the spark programming guide that spark 1.4.1 depends on scala 2.10.x, so I change the build.sbt to:

name := "scalaChartTest"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.4.1"

And the warning disappeared.

Upvotes: 0

Kulu Limpa
Kulu Limpa

Reputation: 3541

You have indirect dependencies to the libraries that are mentioned in the warning. There is a conflict, since the version in the indirect dependency is different from what is specified in your sbt-file (in this case probably by the scala version). The conflict is automatically resolved by sbt (chosing either of the versions depending on your configuration). However, the version that is chosen automatically may not be the version you intend to use, hence the warning.

In your case, this probably isn't an issue. Though, if you want to, you can explicitly exclude the indirect dependencies: 

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.4.1" excludeAll (
  ExclusionRule(organization = "org.scala-lang"),
  ExclusionRule("jline", "jline"),
  ExclusionRule("org.slf4j", "slf4j-api")
  )

Upvotes: 9

Related Questions