CODEWITHSUNDEEP
apache-sparkpyspark
Michael
Michael

Reputation: 42050

How to print out Spark connection of Spark session ?

Suppose I run pyspark command and got global variable spark of type SparkSession. As I understand, this spark holds a connection to the Spark master. Can I print out the details of this connection including the hostname of this Spark master ?

Upvotes: 4

Views: 2066

Answers (1)

Alper t. Turker
Alper t. Turker

Reputation: 35229

For basic information you can use master property:

spark.sparkContext.master

To get details on YARN you might have to dig through hadoopConfiguration:

hadoopConfiguration = spark.sparkContext._jsc.hadoopConfiguration()
hadoopConfiguration.get("yarn.resourcemanager.hostname")

or 

hadoopConfiguration.get("yarn.resourcemanager.address")

When submitted to YARN Spark uses Hadoop configuration to determine the resource manger so these values should match ones present in configuration placed in HADOOP_CONF_DIR or YARN_CONF_DIR.

Upvotes: 4

Related Questions