CODEWITHSUNDEEP
apache-spark
Harshil.Chokshi
Harshil.Chokshi

Reputation: 681

How to listen for job progress in Spark without Spark Rest API

I have created my own spark listener class that extends SparkListener. I have overrided the methods onJobStart and onJobEnd. But can't find a way to get job progress. I want to see the percentage of the job that is complete. I know I can get through Spark Rest API, but unfortunately I can't use that. Is there any other way I can get job progress percentage?

Upvotes: 2

Views: 718

Answers (1)

Igor Berman
Igor Berman

Reputation: 1532

Every job consists of stages, so job start event has stageInfos. Every stage has numTasks, so potentially you can count how many tasks are there in total, and with listening on onTaskEnd/onStageCompleted you can follow job progress.

Upvotes: 2

Related Questions