CODEWITHSUNDEEP
pythonpyspark
james pow
james pow

Reputation: 346

Do I need to stop spark after creating sparksession using pyspark?

So I have this:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("NewSpark").getOrCreate()

I heard you need to stop spark once you're done but is this necessary in my case since it's just a python program?

Upvotes: 0

Views: 3878

Answers (1)

0x5453
0x5453

Reputation: 13599

The session will be closed if the spark object gets destroyed or if the script exits. So you shouldn't need to worry about "dangling connections" or anything like that.

However, if you have a bunch of non-spark work that you want to do at the end of the script, it may still be a good idea to stop the session early to avoid holding that connection open.

Note that you can use the SparkSession object as a context manager to automatically stop it at the end of a scope:

with SparkSession.builder.appName("NewSpark").getOrCreate() as spark:
    # do stuff
    # ...

# spark.stop() gets called automatically here

Upvotes: 3

Related Questions