CODEWITHSUNDEEP
apache-spark
Gokulraj
Gokulraj

Reputation: 520

Retain spark node history

How to retain spark worker and master node history such as completed applications , completed drivers in a cluster. When there is a restart all these history are lost. Is there any specific config to enable for maintaining the history.

Enabled spark event log in spark-defaults.conf

    spark.eventLog.enabled    true
    spark.eventLog.dir   file:////app/spark/logs/data/event_log_dir

But still unable to retain the history

Upvotes: 0

Views: 231

Answers (2)

Aleh Pranovich
Aleh Pranovich

Reputation: 361

Spark UI is available only while application is running. There is a Spark History Server tool, that allows you to see the UI after the application is finished.

More information is in Spark documentation: Spark: Monitoring and Instrumentation - Viewing After the Fact

Upvotes: 0

Serge Harnyk
Serge Harnyk

Reputation: 1339

There is inbox solution - Spark History Server

https://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact

Upvotes: 1

Related Questions