CODEWITHSUNDEEP
apache-sparkhadoop-yarn
Xingjun Wang
Xingjun Wang

Reputation: 423

Application execution monitoring for Spark job on yarn

I can see the application execution information in detail on the Web UI in Spark standalone mode, but when comes to yarn, it is gone. So, where can I see the execution information when job is ran on yarn?

Upvotes: 0

Views: 454

Answers (2)

user1314742
user1314742

Reputation: 2924

You need to configure spark history server with yarn ,and then start it

in your spark-defaults.conf file add the following properties,

spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://LOCATION/TO/SPARK/EVENT/LOG

spark.yarn.historyServer.address        SPARK_HISTORY_SERVER_HOST
spark.history.ui.port                   SPARK_HISTORY_SERVER_PORT

spark.yarn.services                     org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.history.fs.logDirectory           hdfs://LOCATION/TO/SPARK/EVENT/LOG

and then start spark history server:

$/PATH/TO/SPARK/sbin/start-history-server.sh


P.S. I assume that Spark is already configured with hadoop/yarn (so you have set the location of configuration files in spark-env.sh)

Upvotes: 1

RoyaumeIX
RoyaumeIX

Reputation: 1977

You can debug your application , but I guess there is no UI dedicated for that.

Upvotes: 0

Related Questions