CODEWITHSUNDEEP
apache-sparkpysparkhadoop-yarnmaster-slaveexecutor
theGreedyLearner
theGreedyLearner

Reputation: 42

How does Application Master, Spark Driver and Spark Context work together?

After reading over this topic through multiple articles and answers, some ideas are still hazy. My understanding of spark execution process is like this.

When a spark job is submitted an application master is created. This application master launches the driver program. The driver program creates the spark context first. Then the driver program runs the main method of our application.

My questions

  1. How does the driver program creates sparkContext? Does it have the code for it natively and does it as a first step or does it go through the application I submitted and creates it when it encounters a line like val sc = new SparkContext(conf).
  2. Who actually talks with the cluster manager to get the resources? Is it the application master, the driver program or the sparkContext? My understanding was it should be Application Master but I have seen some varying responses on this topic.
  3. Once the resources are allocated, is it the driver program that talks with the node managers and launches executors on them? Or is it the spark Context? Who creates the jobs and tasks and runs them?
  4. Might be a beginner question (which I am to this topic), but what the main method exactly mean in this context? A Python code that I execute from a notebook does not have a main method, how does it work in this case?

Upvotes: 0

Views: 119

Answers (0)

Related Questions