CODEWITHSUNDEEP
scalaapache-spark
Avenger
Avenger

Reputation: 877

Spark parallelism with cores

I have started learning spark and while trying to run this example:

package examples

import org.apache.spark.sql.SparkSession

object Test extends App {
  val spark: SparkSession = SparkSession.builder()
    .master("local[2]")
    .appName("SparkByExample")
    .getOrCreate()

  println("First SparkContext:")
  println("APP Name :"+spark.sparkContext.appName)
  println("Deploy Mode :"+spark.sparkContext.deployMode)
  println("Master :"+spark.sparkContext.master)
  println("Default Min parallelism" + spark.sparkContext.defaultMinPartitions)
  println("Default parallelism" + spark.sparkContext.defaultParallelism)

  val sparkSession2: SparkSession = SparkSession.builder()
    .master("local[1]")
    .appName("SparkByExample-test")
    .getOrCreate()

  println("Second SparkContext:")
  println("APP Name :"+sparkSession2.sparkContext.appName)
  println("Deploy Mode :"+sparkSession2.sparkContext.deployMode)
  println("Master :"+sparkSession2.sparkContext.master)
  println("Default Min parallelism" + sparkSession2.sparkContext.defaultMinPartitions)
  println("Default parallelism" + sparkSession2.sparkContext.defaultParallelism)
}

Here i have created two spark Session the first one with two cores and second one with one core, but when i run it, i get two parallelism for both sessions, I do not understand why?

First SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[2]
Default Min parallelism2
Default parallelism2

Second SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[2]
Default Min parallelism2
Default parallelism2

Upvotes: 0

Views: 315

Answers (1)

vaquar khan
vaquar khan

Reputation: 11449

   println("Default parallelism" + sparkSession2.sparkContext.defaultParallelism)

"sparkContext.defaultParallelism" is returns default level of parallelism defined on
SparkContext ,its default value calculated based on no of cores available on your application.

Upvotes: 1

Related Questions