CODEWITHSUNDEEP
mavenhadoopapache-spark
user1189851
user1189851

Reputation: 5041

Maven could not resolve dependencies spark

I try to build a simple java program: JavaWordCount for spark-1.1.0. 

I get this error: Building JavaWordCount 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.279 s
[INFO] Finished at: 2014-10-23T11:28:30-04:00
[INFO] Final Memory: 9M/156M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project JavaWordCount: Could not resolve dependencies for     project spark.examples:JavaWordCount:jar:1.0-SNAPSHOT: Failure to find org.apache.spark:spark-assembly_2.10:jar:1.1.0 in http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

Here are my dependencies from pom.xml

<dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
     </dependency>
    <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-assembly_2.10</artifactId>
    <version>1.1.0</version>
   </dependency>
   <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-examples_2.10</artifactId>
    <version>1.1.0</version>
    </dependency>
 <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.1.0</version>
 </dependency>

It includes spark assembly.

Any ideas would be really appreciated.

Thank you!

Upvotes: 1

Views: 6711

Answers (1)

khmarbaise
khmarbaise

Reputation: 97409

The problem is that the dependency:

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-assembly_2.10</artifactId>
  <version>1.1.0</version>
</dependency>

is not a jar its a pom file only which means you can't define it like this. You can see it in the error message:

Failure to find org.apache.spark:spark-assembly_2.10:jar:1.1.0

which shows that Maven will try to download a jar file. The means you have to define it like this:

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-assembly_2.10</artifactId>
  <version>1.1.0</version>
  <type>pom</type>
</dependency>

But i'm not sure if this will solve all problems. You should take a deep look into the documentation if this is the right path.

Update: You can also use that as BOM via:

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-assembly_2.10</artifactId>
  <version>1.1.0</version>
  <type>pom</type>
  <scope>import</scope>
</dependency>

Upvotes: 2

Related Questions