CODEWITHSUNDEEP
scalaintellij-ideaapache-sparkscala-2.10apache-spark-1.5
Omid Ebrahimi
Omid Ebrahimi

Reputation: 1190

How to work with Apache Spark using Intellij Idea?

I want to know what is the best way to work with Apache Spark using Intellij Idea? (specially for Scala programming language)

Please explain step-by-step if you can.

Thanks for answer

Upvotes: 4

Views: 4088

Answers (3)

ap-osd
ap-osd

Reputation: 2834

  1. Setup a Scala development environment with IntelliJ. See Scala - Getting Started.

    • JDK is required since Scala is a JVM language
    • sbt is the build tool
    • IntelliJ can be the IDE

  2. To the Scala environment, add the Spark dependency. See Spark - Getting Started.

    • Execute the application using spark-submit

The links provide simple working examples which you can extend to write your own application.

Upvotes: 1

Ajay Gupta
Ajay Gupta

Reputation: 3222

There is good tutorial on Setting up the Spark with Scala on Intellij Idea
Tutorial Link

Let me know if you face any issue.

Upvotes: 8

WestCoastProjects
WestCoastProjects

Reputation: 63231

There is likely no free lunch here. I am a Spark contributor in SQL and MLLib areas and have spent untold hours dealing with Intellij and Spark integration. You can google "stackoverflow intellij spark" and that will give an idea.

Follow imAGin's suggestion to look at some of the tutorials. Use the StackOverflow questions and answers (I have put in many myself). You WILL need to invest a lot of time to get this working. And unfortunately it is not a one-time affair. Spark changes constantly - including its dependencies and build files. So it is a difficult and moving target.

Upvotes: 4

Related Questions