Reputation: 61
I try to "run" spark jobs width my java application, searching, I found the following two methods:
ClientsArguments
and SparkLauncher
.
Could someone explain to me the difference between the two? The difference between launch and submit job/application Spark? Thank you.
Upvotes: 0
Views: 157
Reputation: 61
SparkLauncher is just a wrapper library for spark-submit
, it coverts your SparkLauncher
code to spark-submit
script and then trigger the jobs.
The mechanism is the same with spark-submit
script, if you look at the source code of SparkLauncher, it uses ProcessBuilder
to construct the shell.
If you want to use SparkLauncher
, you need to specify $JAVA_HOME
, $SPRAK_HOME
and other essential parameters. There are some limitation for SparkLauncher
, the machine your SparkLauncher
runs must have a $JAVA_HOME
and $SPARK_HOME
(Spark Library) which is used for SparkLauncher
locating the script and related dependencies. This is soft of impossible for some cloud environment like CloudFoundry etc.
You could assume that SparkLauncher
equals spark-submit
script, you could choose client or master, local or yarn mode.
ClientsArguments
is a class for YARN script, which is only works for yarn-mode.
Upvotes: 1