CODEWITHSUNDEEP
apache-spark
Nicola Ferraro
Nicola Ferraro

Reputation: 4189

Why is SparkListenerApplicationStart never fired?

I am writing a Spark application and I need to intercept the status of the running jobs. I implemented a SparkListener for this purpose, using the following code:

class MyAppListener extends SparkListener {

    override def onApplicationStart(ev: SparkListenerApplicationStart): Unit = {
      println("AAA: Application Start")
    }

    override def onApplicationEnd(ev: SparkListenerApplicationEnd): Unit = {
      println("AAA: Application End")
    }
  }
}

Then, I used the following code to start the application and see the events:

val appListener = new MyAppListener
val conf = new SparkConf().setAppName("Listener")
val sc = new SparkContext(conf) 
sc.addSparkListener(appListener)
println(sc.parallelize(1 to 10).count)
sc.stop()

In the logs, I see the string "AAA: Application End", but I don't see the start of the application.

Configuration:

Upvotes: 5

Views: 1708

Answers (1)

Xuan Huy Pham
Xuan Huy Pham

Reputation: 296

You were adding your listener to spark in the wrong place, When you initiate a spark context, it also starts your application.=> At the point you added your listener, onApplicationStart has already been fired.

Solution: Add your listener to SparkConf.

sparkConf.set("spark.extraListeners","your.listener.class")

Upvotes: 12

Related Questions