CODEWITHSUNDEEP
scalaapache-sparkhadoop-yarn
JediKid
JediKid

Reputation: 83

In a scala spark job, running in yarn, how can I fail the job so that yarn shows a Failed status

I have a simple if statement in my scala spark job code, that if false i want to stop the job and mark it failed. I want the yarn UI to show the spark job with a status of failed, but everything i've done so far has stopped the job, but only shows up as successfully finished on the yarn UI.

    if(someBoolen) {
        //context.clearAllJobs()
        //System.exit(-1)
        //etc, nothing so far, stops the job and show as failed in the yarn UI
    }

Any help would be great.

Upvotes: 1

Views: 5627

Answers (1)

puhlen
puhlen

Reputation: 8529

Throwing an exception (and not catching it) will cause the process to fail.

if(someBoolen) {
    throw new Exception("Job failed");
}

Upvotes: 5

Related Questions