CODEWITHSUNDEEP
pythonapache-sparkpyspark
Michael
Michael

Reputation: 51

Print SparkSession Config Options

When I start pyspark, a SparkSession is automatically generated and available as 'spark'. I would like to print/view the details of the spark session but am having a lot of difficulty accessing these parameters.

Pyspark auto creates a SparkSession. This can be created manually using the following code:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("PythonSQL")\ 
        .config("spark.some.config.option", "some-value")\ 
        .getOrCreate()

I would like to view/print the appname and config options. The reason I would like to see these is as a result of another issue that I am experiencing which this may shed light on.

Upvotes: 5

Views: 10015

Answers (1)

zero323
zero323

Reputation: 330073

Application name can be accessed using SparkContext:

spark.sparkContext.appName

Configuration is accessible using RuntimeConfig:

from py4j.protocol import Py4JError

try: 
   spark.conf.get("some.conf")
except Py4JError as e:
   pass

Upvotes: 6

Related Questions