CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Markus
Markus

Reputation: 3782

How to update existing SparkSession instance or create a new one in spark-shell?

When I launch spark-shell, it creates an instance of SparkSession. However, I should create it as follows:

val spark = SparkSession.builder()
                        .config("es.nodes",elasticHost)
                        .config("es.port",elasticPort)
                        .config("es.nodes.wan.only","true")
                        .appName("Test")
                        .getOrCreate()

How can I update the existing spark in spark-shell or create the new one as I showed above?

Upvotes: 6

Views: 15774

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74759

You can set the configuration properties using SparkSession.conf.set or create another SparkSession instance using SparkSession.newSession and then set the properties.

set(key: String, value: String): Unit Sets the given Spark runtime configuration property.

newSession(): SparkSession Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.

Both ways work (almost) the same with the difference that you can temporarily set a property to a new value and use both SparkSessions simultaneously.

// hello property is not set
scala> spark.conf.getOption("hello")
res1: Option[String] = None

scala> spark.conf.set("hello", "world")

// hello property is set
scala> spark.conf.getOption("hello")
res3: Option[String] = Some(world)

// create a new SparkSession (so you'll have two at the same time)
val ns = spark.newSession

// hello is not set in a new session
scala> ns.conf.getOption("hello")
res4: Option[String] = None

ns.conf.set("hello", "hello in another session")

scala> ns.conf.getOption("hello")
res8: Option[String] = Some(hello in another session)

// the value of hello in the initial SparkSession remains unchanged
scala> spark.conf.getOption("hello")
res9: Option[String] = Some(world)

Upvotes: 10

Related Questions