CODEWITHSUNDEEP
scalasbtapache-spark
Y.Prithvi
Y.Prithvi

Reputation: 1221

UNRESOLVED DEPENDENCIES error while trying to create jar

I'm trying to build a Scala jar file to run it in spark.
I'm following this tutorial.
when trying to build jar file using sbt as here, i'm facing with following error

[info] Resolving org.apache.spark#spark-core_2.10.4;1.0.2 ...
[warn]  module not found: org.apache.spark#spark-core_2.10.4;1.0.2
[warn] ==== local: tried
[warn]   /home/hduser/.ivy2/local/org.apache.spark/spark-core_2.10.4/1.0.2/ivys/ivy.xml
[warn] ==== Akka Repository: tried
[warn]   http://repo.akka.io/releases/org/apache/spark/spark-core_2.10.4/1.0.2/spark-core_2.10.4-1.0.2.pom
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10.4/1.0.2/spark-core_2.10.4-1.0.2.pom
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.spark#spark-core_2.10.4;1.0.2: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[error] {file:/home/prithvi/scala/asd/}default-d57abf/*:update: sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.10.4;1.0.2: not found
[error] Total time: 2 s, completed 13 Aug, 2014 5:24:24 PM

what's the issue and how to solve it.

Dependency issue has been resolved. Thank you "om-nom-nom"
but new error arised

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::              FAILED DOWNLOADS            ::
[warn]  :: ^ see resolution messages for details  ^ ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit
[warn]  :: org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit
[warn]  :: org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit
[warn]  :: org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[error] {file:/home/prithvi/scala/asd/}default-c011e4/*:update: sbt.ResolveException: download failed: org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit
[error] download failed: org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit
[error] Total time: 855 s, completed 14 Aug, 2014 12:28:33 PM

Upvotes: 21

Views: 22585

Answers (5)

Danylo Zherebetskyy
Danylo Zherebetskyy

Reputation: 1517

I had the same issue. Looks like that some bugs are in different versions/compilations/etc.

For me the following build.sbt worked fine

name := "My Project"  
version := "1.0"  
scalaVersion := "2.11.8"  
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.2"

Hope it helps

Upvotes: 3

imonaboat
imonaboat

Reputation: 29

How can you change the current dependencies? I mean, when you type sbt package for a build file like: 

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"

SBT will start resolving and downloading all kinds of dependencies. But if you see that it is failing on a dependency that is no longer inthe maven repo, what to do? Where can you change the dpencies it tries.

@OP: The problem is that your SBT is outdated. If you downloaded it using apt, you can use apt to remove it as well. In any case, download the latest .tgz (not the .deb) and simply unpack it, after that add the /sbt/bin/ folder to your .bashrc I noticed that older SBT's (the .deb and apt-get versions) work with older scala versions. You either need to manually add or change the dependencies that the older SBT is trying to find or simply change to the latest (not sooo)SBT.

Upvotes: 0

yang jun
yang jun

Reputation: 153

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.1.0",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016",
"org.eclipse.jetty.orbit" % "javax.transaction" % "1.1.1.v201105210645",
"org.eclipse.jetty.orbit" % "javax.mail.glassfish" % "1.4.1.v201005082020"

)

Upvotes: 0

nat
nat

Reputation: 566

spark-core_2.10.4;1.0.2 means that it is build on top of scala 2.10 vesion. so you have to specified this scalaVersion := "2.10.4" in your build file. Please check your .sbt file and change accordingly.

Upvotes: 0

om-nom-nom
om-nom-nom

Reputation: 62855

You have your dependency defined as

"org.apache.spark" %% "spark-core" % "1.0.2"

That %% instructs sbt to substitute current scala version to artifact name. Apparently, spark was build for the whole family of 2.10 scala, without specific jars for 2.10.1, 2.10.2 ...

So all you have to do is to redefine it as: 

"org.apache.spark" % "spark-core_2.10" % "1.0.2"

Upvotes: 43

Related Questions