CODEWITHSUNDEEP
javascalaapache-spark
St.Antario
St.Antario

Reputation: 27375

Why does spark kill tasks?

I'm running a spark computational application and I regularly run into some issue with task killing. Here is how it looks like in my spark console:

enter image description here

As can be seen there are some jobs with the description (_num_ killed: another attempt succeeded). This is not just failed, this is something different. Can someone explain what is it?

Upvotes: 3

Views: 5417

Answers (2)

Joe K
Joe K

Reputation: 18424

If a task appears to be taking an unusually long time to complete, Spark may launch extra duplicate copies of that task in case they can complete sooner. This is referred to as speculation or speculative execution. If one copy succeeds, the others can be killed.

See the parameters starting with spark.speculation here: https://spark.apache.org/docs/latest/configuration.html

Upvotes: 11

vaquar khan
vaquar khan

Reputation: 11449

Killed - It means that that executor is killed by an the Worker who stopped and asked to kill the executor. This situation can be because of many reasons like by some user driven action or may be your executor finished processing but due to some reason it does not exists but worker is exiting so it needs to Kill the executor.

Upvotes: 1

Related Questions