CODEWITHSUNDEEP
scalaapache-spark
Hlib
Hlib

Reputation: 3064

How to get number of cores used in current job?

SparkContext has getExecutorMemoryStatus method. But it's for memory status of an Executor.

Is there any way how to get core status? I use Spark Standalone Cluster.

Upvotes: 4

Views: 3332

Answers (1)

Ram Ghadiyaram
Ram Ghadiyaram

Reputation: 29155

Out of below probably option 3 is the one you are looking for..

  • Option 1 : Spark web ui gives me the information about total cores and used cores.

enter image description here

  • Option 2 : Defaults :

    sc.defaultParallelism typically set to the number of worker cores in your cluster

  • Option 3 : Can use ExectorInfo.totalCores like below and try... it should work.

docs says

public class ExecutorInfo extends Object Stores information about an executor to pass from the scheduler to SparkListeners.

import org.apache.spark.scheduler.{SparkListener, SparkListenerExecutorAdded}

/**
  * Logs info of added executors.
  */
final class ExecutorLogger extends SparkListener {

  override def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit =
    println(s"\rExecutor ${executorAdded.executorId} added: ${executorAdded.executorInfo.executorHost} ${executorAdded.executorInfo.totalCores} cores")

}

Upvotes: 0

Related Questions