Reputation: 1190
I want to know what is the best way to work with Apache Spark using Intellij Idea? (specially for Scala programming language)
Please explain step-by-step if you can.
Thanks for answer
Upvotes: 4
Views: 4088
Reputation: 2834
Setup a Scala development environment with IntelliJ. See Scala - Getting Started.
sbt
is the build toolTo the Scala environment, add the Spark dependency. See Spark - Getting Started.
spark-submit
The links provide simple working examples which you can extend to write your own application.
Upvotes: 1
Reputation: 3222
There is good tutorial on Setting up the Spark with Scala on Intellij Idea
Tutorial Link
Let me know if you face any issue.
Upvotes: 8
Reputation: 63231
There is likely no free lunch here. I am a Spark contributor in SQL and MLLib areas and have spent untold hours dealing with Intellij and Spark integration. You can google "stackoverflow intellij spark" and that will give an idea.
Follow imAGin's suggestion to look at some of the tutorials. Use the StackOverflow questions and answers (I have put in many myself). You WILL need to invest a lot of time to get this working. And unfortunately it is not a one-time affair. Spark changes constantly - including its dependencies and build files. So it is a difficult and moving target.
Upvotes: 4