CODEWITHSUNDEEP
apache-sparkhadoop-yarn
matthieun
matthieun

Reputation: 833

What is the entry point of Spark container in YARN cluster mode?

What is the main entry point of a Spark executor when a Spark job is being run in Yarn cluster mode (for Spark 1.2.0+)?

What I am looking for is the Scala class name for the entry point of an Executor (which will be the process executing one of the tasks on a slave machine).

Upvotes: 1

Views: 1478

Answers (2)

Jacek Laskowski
Jacek Laskowski

Reputation: 74619

I think what you're asking about is org.apache.spark.executor.Executor or perhaps org.apache.spark.executor.Executor$TaskRunner. It is TaskRunner that will ultimately run a task.

It is regardless of the deploy mode (client vs cluster) or the cluster manager used, i.e. Hadoop YARN or Spark Standalone or Apache Mesos.

Upvotes: 2

Justin Pihony
Justin Pihony

Reputation: 67065

spark-submit --class [FULLY QUALIFIED CLASS NAME]
             --master yarn-cluster
             [JAR_TO_USE]

So, given the above, the class to be used is the one specified, which is loaded from the given jar, and it searches within that class for a static main method.

From SparkSubmit.scala:

val mainMethod = mainClass.getMethod("main", new Array[String](0).getClass)

Upvotes: 2

Related Questions