CODEWITHSUNDEEP
apache-spark
grepIt
grepIt

Reputation: 116

How to identify if the JVM thats running is Driver or Executor in Apache Spark

I am trying to extend Spark logging to Elasticsearch. And I am looking for a way to identity whether this message is logged by the driver or from executors while building the log statement.

Any suggestions please?

Upvotes: 0

Views: 238

Answers (1)

Chitral Verma
Chitral Verma

Reputation: 2855

I'm going to give you the example using the following code to differentiate between driver and executor,

object test{
    def main(args:Array[String]){
        log.info("This is driver")
        val sparkSession = ???

        spark.read
        ...
        .map(x=>{
            ...
            log.info("This is executor")
            ...
            })

        log.info("This is driver")
    }
}

As you can see that anything in between map/ foreach and other transformations/ actions is executed on executors, everything else is executed on driver.

Hope this clarifies.

Upvotes: 1

Related Questions