CODEWITHSUNDEEP
apache-sparkhortonworks-data-platformazure-hdinsight
aaronsteers
aaronsteers

Reputation: 2571

Configure (no) retries for Spark job in Livy submit

When submitting a long-running Spark batch job through Livy, the job is defaulting to five retries, which takes forever to finally fail. How can I change this so the job fails immediately?

My environment is Spark 1.6, running on Azure HDInsight (HDP).

Thanks!

Upvotes: 0

Views: 615

Answers (1)

Lin Chan
Lin Chan

Reputation: 46

This is a configuration on your yarn, not on Livy. Go to yarn's configuration page under "Advanced yarn-site" and change "yarn.resourcemanager.am.max-attempts" from 5 to 1 if you want it to do no retry.

Upvotes: 3

Related Questions