CODEWITHSUNDEEP
apache-spark
Adam Bacon
Adam Bacon

Reputation: 55

What part of Spark return failed task to a different node?

When a task running on a node fails, Spark will automatically return the task to a different node. My Questions are:

Upvotes: 1

Views: 280

Answers (1)

Harjeet Kumar
Harjeet Kumar

Reputation: 524

When a spark task is failed following things happen

  1. Node manager at that machine try to rerun that task on same machine and also informs APPLICATION MASTER.
  2. Based on speculative execution, application master may decide to run a duplicate task on another machine. Resource Manager does not restart the task
  3. Task is restarted from beginning. Since all partitioned processed by that task were lost in that failure that happened earlier. Thats where spark RDD lineage comes in picture. spark takes spark lineage and tries to recreate partition which was lost as part of task failure.

Upvotes: 2

Related Questions