CODEWITHSUNDEEP
apache-sparkpyspark
sds
sds

Reputation: 60014

Can I change SparkContext.appName on the fly?

I know I can use SparkConf.set('spark.app.name',...) to set appName before creating the SparkContext.

However, I want to change the name of the application as it progresses, i.e., after SparkContext has been created.

Alas, setting sc.appName does not change how the job is shown by yarn application -list.

Is there a way?

Upvotes: 12

Views: 10821

Answers (1)

Svend
Svend

Reputation: 7180

This is not possible: any update to the sparkConf, including spark.app.name, is only taken into account before the instance of SparkConf is used to instanciate a SparkContext:

Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.

https://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.SparkConf

Upvotes: 19

Related Questions