CODEWITHSUNDEEP
pythonapache-sparkpyspark
Tanner Clark
Tanner Clark

Reputation: 691

What is the Difference between SparkSession.conf and SparkConf?

I understand that you create a SparkSession from a SparkConf object but does that mean the SparkSession.conf is the same as SparkConf()?

Upvotes: 0

Views: 279

Answers (1)

Naga
Naga

Reputation: 426

Yes, if (SparkSession.builder.config(conf=SparkConf())) you create your SparkSessioin with SparkConf object

You can confirm this from PySpark source code

here is the code for SparkSession.conf , which returns self._conf and if you back track this you can see that it's getting set or created in getOrCreate method link

In getOrCreate method you can observe that SparkConf set with options that are passed at the creation time of SparkSession

enter image description here

You can further check that self._options set with values in config method link which in our case passing SparkConf object's (key, value)

enter image description here

Upvotes: 2

Related Questions