CODEWITHSUNDEEP
apache-spark
fish748
fish748

Reputation: 183

How can I tell if a Spark job is successful or not?

I want to do some clean-ups after a Spark job is finished, and the action I would do depends on whether the job is successful or not.

So how where can I get the result for the job?

I already tried the SparkListener, it worked fine when the job is successful, but if the job fails, the listener seems not called.

Upvotes: 1

Views: 1301

Answers (1)

Yehor Krivokon
Yehor Krivokon

Reputation: 877

You can use REST API of the Spark HistoryServer to check the state of the job.
Here you can find all information that you need:
https://spark.apache.org/docs/latest/monitoring.html

Upvotes: 1

Related Questions