CODEWITHSUNDEEP
apache-sparkamazon-emr
manisha
manisha

Reputation: 31

Spark jobs running on EMR cluster. system.exit(0) used to gracefully completion of job but still Step on EMR fails

In spark job. I am using if file not found the system.exit(0). It should gracefully complete the job. Locally It is successfully completed. But when I am running on EMR. Step is failing.

Upvotes: 3

Views: 3030

Answers (1)

Ivan Vergiliev
Ivan Vergiliev

Reputation: 3841

EMR uses YARN for cluster management and launching Spark applications. So when you're running a Spark app with --deploy-mode: cluster in EMR, the Spark application code is not running in a JVM on its own, but is rather executed by the ApplicationMaster class.

Browsing through the ApplicationMaster code can explain what happens when you try to execute System.exit(). The user application is launched in startUserApplication, and then the finish method is called after the user application returns. However, when you call System.exit(0), what is executed instead is a shutdown hook which sees that your code hasn't finished successfully, and marks it as an EXIT_EARLY failure. It also has this useful comment:

  // The default state of ApplicationMaster is failed if it is invoked by shut down hook.
  // This behavior is different compared to 1.x version.
  // If user application is exited ahead of time by calling System.exit(N), here mark
  // this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call
  // System.exit(0) to terminate the application.

Upvotes: 8

Related Questions