CODEWITHSUNDEEP
sbtapache-spark
Niko Gamulin
Niko Gamulin

Reputation: 66565

Spark unresolved dependencies hadoop

I tried to build self-contained example scala application, but when running sbt package I get the following:

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.hadoop#hadoop-yarn-common;1.0.4: not found
[warn]  :: org.apache.hadoop#hadoop-yarn-client;1.0.4: not found
[warn]  :: org.apache.hadoop#hadoop-yarn-api;1.0.4: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[error] {file:/home/niko/workspace/Spark/recommender/}default-3ebb80/*:update: sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-yarn-common;1.0.4: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-yarn-client;1.0.4: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-yarn-api;1.0.4: not found

Does anyone know what has to be configured in order to run the app successfully (if possible without hadoop installed)?

Thanks!

Upvotes: 1

Views: 2424

Answers (4)

Chris
Chris

Reputation: 1742

i solved it by enabling the auto-import option and then compiling it again

Upvotes: 0

胡昌新
胡昌新

Reputation: 21

The problem is because your sbt fails to retrieve target file, the URL for repos is not unavailable.

Download the latest sbt version, and add the following to ~/.sbt/repositories:

[repositories]
  local
  sbt-releases-repo: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  sbt-plugins-repo: http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  maven-central: http://repo1.maven.org/maven2/

Upvotes: 2

Michael
Michael

Reputation: 1

I had the exact same problem when trying to get my spark program to build. I found out that my version of sbt was giving me the errors. I would recommend completely removing sbt. Then download it from here http://www.scala-sbt.org/download.html. Download the .tgz. Extract it to your home folder. Then add the bin directory to your path.

Upvotes: 0

Sean Owen
Sean Owen

Reputation: 66886

You enabled the YARN profile, but did not set hadoop.version. The default Hadoop version is 1.0.4, and there is no such YARN. In general you want to specify hadoop.version no matter what.

Upvotes: 1

Related Questions