CODEWITHSUNDEEP
javahadoopclouderaapache-spark
mgurov
mgurov

Reputation: 63

Remote running spark job

Trying to run start Spark job.

Spark installed as parcel inside Cloudera Hadoop.

How to start Spark job remotely with Java API?

Upvotes: 1

Views: 2928

Answers (2)

Rich
Rich

Reputation: 2885

To submit Spark applications programmatically from code as opposed to using the submit scripts, you will need to create a SparkContext.

Here is the Java API SparkContext

To configure the context further you provide it a SparkConf with values that match entries on the configuration page: Configuration

Your cluster will need a copy of the code you are going to submit to it in its classpath. There are multiple ways to do this, you can manage it on the cluster manually or pass it in to the SparkConf with the setJars method.

As of Spark 1.3.1, you can only have 1 SparkContext created at a time per JVM. So if you want to submit concurrent applications from the same JVM you will need to manage this shared resource properly.

Upvotes: 1

hnahak
hnahak

Reputation: 246

Check your spark-cluster from URl spark-master-hostname: if it is running fine go to the machine from where you want to launch the job (>> this machine should also has installed spark library to submit the job And) hit this command from you machine spark-submit --class --jars --master spark://:7077 .jar

Upvotes: -1

Related Questions