CODEWITHSUNDEEP
apache-spark
Sourav Hazra
Sourav Hazra

Reputation: 43

How does DAG make Apache Spark fault-tolerant?

I am a beginner to Apache Spark. I was trying to understand the concept of DAG which Apache Spark creates and when we apply transformations one after another and which gets executed once an action is performed.

What I could make out is that in the event of a job failure, DAG comes to the rescue. Since all the intermediate RDDs are stored in the memory, Spark knows till which step the job ran successfully and restart the job from that point only, instead of starting the job from the beginning.

Now I have several questions here:

  1. Can DAG make Spark resilient to node failures ?
  2. Is it the driver node which maintains the DAG ?
  3. Can there be multiple DAGs for a single execution ?

Upvotes: 2

Views: 1119

Answers (1)

Nikunj Kakadiya
Nikunj Kakadiya

Reputation: 2998

  • What I could make out is that in the event of a job failure, DAG comes to the rescue. Since all the intermediate RDDs are stored in the memory, Spark knows till which step the job ran successfully and restart the job from that point only, instead of starting the job from the beginning.

I think what you have said above based on your understanding is not fully correct.

  1. DAG comes to rescue in the event of Node failures and not the job failures.
  2. Spark driver knows which worker node is working on which partition of data through cluster manager. So, when the cluster manager comes to know that specific node is dead then it assigns another node to start processing. Because of the DAG , new worker node know the tasks that it has to work on but it has to perform all the transformation from the start. If the node fails all the stuffs that you had in memory also goes away. DAG helps spark to be fault-tolerant because it can recover from node failures.

Your Question 1: Can DAG make Spark resilient to node failures ?

Yes DAG makes it fault tolerance to node failures.

Question 2: Is it the driver node which maintains the DAG ?

Yes. When an action is called, the created DAG is submitted to DAG Scheduler, where it gets converted into stages of jobs.

Question 3: Can there be multiple DAGs for a single execution ?

No. you cannot have multiple DAGs because DAG is kind of a graph that represents the operation that you perform.

Upvotes: 3

Related Questions