CODEWITHSUNDEEP
python-2.7apache-sparkpysparkhadoop-yarn
Michael
Michael

Reputation: 1428

How to relaunch a Spark executor after it crashes (in YARN client mode)?

Is it possible to relaunch a Spark executor after it crashes? I understand that the failed tasks are re-run in the existing working Spark executors, but I hope there is a way to relaunch the crashed Spark executor.

I am running pyspark 1.6 on YARN, in client mode

Upvotes: 2

Views: 2728

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74619

No. It is not possible. Spark takes care of it and when an executor dies, it will request a new one the next time it asks for "resource containers" for executors.

If the executor was close to the data to process Spark will request for a new executor given locality preferences of the task(s) and chances are that the host where the executor has died will be used again to run the new one.

An executor is a JVM process that spawns threads for tasks and honestly does not do much. If you're concerned with the data blocks you should consider using Spark's external shuffle service.

Consider reading the document Job Scheduling in the official documentation.

Upvotes: 2

Related Questions