CODEWITHSUNDEEP
scalahadoopapache-sparkintellij-ideabigdata
TomG
TomG

Reputation: 291

Run Spark code written in Scala in spark cluster

I have IntelliJ IDE installed in my laptop. I am trying to do some Bigdata Spark POCs written in Scala. My requirement is that the spark-scala code written in IntelliJ IDE should run in spark cluster when I click Run. My spark cluster is residing in windows azure cloud. How can I achieve this?

Upvotes: 2

Views: 828

Answers (2)

sai pradeep kumar kotha
sai pradeep kumar kotha

Reputation: 206

step 1:before starting the process you have to download the hadoop bin

https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin

and you have to set the hadoop home in environment variables example:C:\Hadoop\hadoop

Step2:Then Download the spark of desired version

add the path C:\Hadoop\spark-1.6.0-bin-hadoop2.6\bin to environment variables

step3: open cmd and go to spark folder till bin C:\Hadoop\spark-1.6.0-bin-hadoop2.6\bin and type following command spark-class org.apache.spark.deploy.master.Master it will give the spark master ip like example spark://localhost:7077 step4:open another cmd and go to spark folder till bin and type following command spark-class org.apache.spark.deploy.worker.Worker SparkMasterIp

step5: To check it is working or not we can test by below command C:\Hadoop\spark-1.6.0-bin-hadoop2.6\bin\spark-shell -master masterip

now you can build your jar and submit the jar to spark-submit from cmd

Upvotes: 0

koiralo
koiralo

Reputation: 23109

One way is to create a script to run the jar file created, and run that script.

And another way it touse Azure Toolkit plugin.

You can use Azure Toolkit for IntelliJ Intellij Idea plugin to submit, run debug the spark application

Search and install the plugin as below

enter image description here

To submit and run the application you can follow the documentation here

https://azure.microsoft.com/en-us/blog/hdinsight-tool-for-intellij-is-ga/

Here is the example https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-intellij-tool-plugin

Hope this helps!

Upvotes: 2

Related Questions