CODEWITHSUNDEEP
scalaapache-sparkpyspark
Sangram Gaikwad
Sangram Gaikwad

Reputation: 804

Apache Spark: Disable Spark Web UI in Spark Standalone mode

I am running Apache Spark 2.1.1 in Standalone Mode with client deploy mode. I want to disable Spark web UI for master and all workers.

Refered to: https://spark.apache.org/docs/latest/configuration.html#spark-ui and Used following conf in $SPARK_HOME/conf/spark-defaults.conf:

spark.ui.showConsoleProgress=false
spark.ui.enabled=false
spark.ui.killEnabled=false

Yet I could still see web UI at default port 8080 (for Spark Master and 8081 for Spark Worker). I have the same configuration in all the worker nodes.

Upvotes: 4

Views: 2607

Answers (1)

Jeremie Basso
Jeremie Basso

Reputation: 1

It seems that their is no way to disable Spark Master UI.

However, you can change the port it is running on with /spark-master.sh -p PORT as described here : https://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts

Finaly, you could build a script that kill the process using the specific port after the spark-master.sh have run.

Hope it helps a bit.

Upvotes: 0

Related Questions