CODEWITHSUNDEEP
apache-sparkjupyter-notebook
wally
wally

Reputation: 21

how to call spark.stop() old session?

In Jupyter, I created spark session like this

spark = SparkSession.builder
         .master("yarn")
         .appName("test procjet")
         .getOrCreate()

But I forgot spark.stop() and shutdown Jupyter notebook. I want to call stop() to old session so I tried getOrCreate() but spark history server show new application.

How can I get old spark session and stop? I also tried yarn application -kill <appid> but spark history still show that.

Upvotes: 2

Views: 1293

Answers (1)

Ani Menon
Ani Menon

Reputation: 28209

When a Jupyter Notebook is shutdown, sparkSession used in that is also stopped as Jupyter is the client which created it. The same is the case when you forget to close() SparkSession in a script which uses it.

Upvotes: 1

Related Questions