CODEWITHSUNDEEP
apache-spark
techagrammer
techagrammer

Reputation: 1306

How to build and debug spark from IDE(preferrably Eclipse)?

I wanted to contribute to spark.

I cloned the git repository locally. Please suggest how to setup spark first and then run a hello world over it from IDE itself.

Upvotes: 3

Views: 1410

Answers (2)

Iulian Dragos
Iulian Dragos

Reputation: 5712

I assume you already have the latest release of Scala IDE (4.0 at this point) from scala-ide.org.

  • export projects using sbt eclipse, I guess you figured that out already.
  • import all projects in your workspace (Import Existing projects)
    • you will probably see a number of errors related to "cross-compiled libraries"

If you want to develop on Scala 2.10, you need to configure a Scala installation for the exact Scala version that’s used to compile Spark. At the time of this writing that is Scala 2.10.4.

  • you can do that in Eclipse Preferences -> Scala -> Installations by pointing to the lib/ directory of your Scala 2.10.4 distribution.
  • select all Spark projects and right-click, choose Scala -> Set Scala Installation and point to the 2.10.4 installation. This should clear all errors about invalid cross-compiled libraries.
  • a clean build should succeed.

You can easily find examples on getting started with Spark, for example here. You can run a Spark app using right-click -> Run As Scala Application.

Upvotes: 1

Ashrith
Ashrith

Reputation: 6855

For importing/building Spark in IntelliJ or Eclipse follow this guide.

If you are interested in contributing to Spark visit this wiki page for more information:

Upvotes: 1

Related Questions