CODEWITHSUNDEEP
apache-spark
user2552806
user2552806

Reputation:

What is the difference between Spark executor states Exited vs Killed?

I`m working with spark 1.2.1. When I run spark jobs sometimes I get executor state "Exited" and sometimes "Killed", in both scenarios the job is finished successfully and I invoking SparkContext.stop()...

I failed to understand the meaning of those states.

enter image description here

enter image description here

What is the difference between Spark executor states Exited vs Killed?

Upvotes: 3

Views: 2849

Answers (1)

Sumit
Sumit

Reputation: 1420

Exited - It means that the Executor finished the processing and it exists cleanly without any errors or exception.

Killed - It means that that executor is killed by an the Worker who stopped and asked to kill the executor. This situation can be because of many reasons like by some user driven action or may be your executor finished processing but due to some reason it does not exists but worker is exiting so it needs to Kill the executor.

Also as a good practice we should invoke SparkContext.stop() method at the end of Job. Though this would not ensure that you will always have "Exited" status but it will definitely ensure that cleanup is executed and resources are de-allocated.

Upvotes: 3

Related Questions