CODEWITHSUNDEEP
apache-sparkhadoop-yarn
user401445
user401445

Reputation: 1014

How to control Spark application per task/stage/job attempt?

I'd like to stop Spark from retrying a Spark application in case some particular exception is thrown. I only want to limit the number of retries in case certain conditions are met. Otherwise, I want default number of retries.

Note that there is only one Spark job which a Spark application runs.

I tried setting javaSparkContext.setLocalProperty("spark.yarn.maxAppAttempts", "1"); in the case of the exception, but it still retries the whole job.

I submit the Spark application as follows:

spark-submit --deploy-mode cluster theSparkApp.jar

I have a use case where I want to delete the output if it is created by the previous retry of the same job, but fail the job if the output folder is not empty (in 1st retry). Can you think of any other way to achieve this?

Upvotes: 3

Views: 2795

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74779

I have a use case where I want to delete the output if it is created by the previous retry of the same job, but fail the job if the output folder is not empty (in 1st retry). Can you think of any other way to achieve this ?

You can use TaskContext to control how your Spark job behaves given, say, the number of retries as follows:

val rdd = sc.parallelize(0 to 8, numSlices = 1)

import org.apache.spark.TaskContext

def businessCondition(ctx: TaskContext): Boolean = {
  ctx.attemptNumber == 0
}

val mapped = rdd.map { n =>
  val ctx = TaskContext.get
  if (businessCondition(ctx)) {
    println("Failing the task because business condition is met")
    throw new IllegalArgumentException("attemptNumber == 0")
  }
  println(s"It's ok to proceed -- business condition is NOT met")
  n
}
mapped.count

Upvotes: 1

Related Questions