CODEWITHSUNDEEP
scalamavenapache-sparkmahout
HHH
HHH

Reputation: 6465

How to build spark application using Scala IDE and Maven?

I'm new to Scala, Spark and Maven and would like to build spark application described here. It uses the Mahout library.

I have Scala IDE install and would like to use Maven to build the dependencies (which are the Mahout library as well as Spark lib). I couldn't find a good tutorial to start. Could someone help me figure it out?

Upvotes: 3

Views: 12813

Answers (2)

saurzcode
saurzcode

Reputation: 837

You need following tools to get started ( based on recent availability) -

  • Scala IDE for Eclipse – Download latest version of Scala IDE from here.

  • Scala Version – 2.11 ( make sure scala compiler is set to this version as well)

  • Spark Version 2.2 ( provided in maven dependency)

  • winutils.exe

For running in Windows environment , you need hadoop binaries in windows format. winutils provides that and we need to set hadoop.home.dir system property to bin path inside which winutils.exe is present. You can download winutils.exe here and place at path like this – c:/hadoop/bin/winutils.exe

And, you can define Spark Core Dependency in your Maven POM.XML for your project, to get started with.

   <dependency> <!-- Spark dependency -->
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-core_2.11</artifactId>
     <version>2.2.0</version>
     <scope>provided</scope>
   </dependency>

And in your Java/Scala class define this property, to run on your local environmet on Windows - 

System.setProperty("hadoop.home.dir", "c://hadoop//");

More details and full setup details can be found here.

Upvotes: 0

suztomo
suztomo

Reputation: 5202

First try compiling simple application with Maven in Scala IDE. The key of Maven project is directory structure and pom.xml. Although I don't use Scala IDE, this document seems helpful. http://scala-ide.org/docs/tutorials/m2eclipse/

Next step is to add dependency on Spark in pom.xml you can follow this document. http://blog.cloudera.com/blog/2014/04/how-to-run-a-simple-apache-spark-app-in-cdh-5/

For latest version of Spark and Mahout artifacts you can check them here: http://mvnrepository.com/artifact/org.apache.spark http://mvnrepository.com/artifact/org.apache.mahout

Hope this helps.

Upvotes: 3

Related Questions