CODEWITHSUNDEEP
apache-sparkpyspark
Bob
Bob

Reputation: 561

How can I find the value of specific Spark configuration property?

How can I find the value of a spark configuration in my spark code?

For example, I would like to find the value of spark.sql.shuffle.partitions and reference this in my code.

The following code will return all values:-

spark.sparkContext.getConf().getAll()

How can I retrieve a single configuration setting?

Upvotes: 9

Views: 7009

Answers (1)

thePurplePython
thePurplePython

Reputation: 2767

Like this.

spark.conf.get("spark.sql.shuffle.partitions")
'200' # returns default value here

Upvotes: 15

Related Questions