CODEWITHSUNDEEP
apache-spark
neeraj
neeraj

Reputation: 900

what is "spark.history.retainedApplications" points to

As per apache doc "http://spark.apache.org/docs/latest/monitoring.html"

spark.history.retainedApplications points to "The number of application UIs to retain. If this cap is exceeded, then the oldest applications will be removed"

But I see more than configured apps into the UI. Is it correct or it stores those many apps into memory only or load again into memory when needed. Please clarify. Thx

Upvotes: 1

Views: 943

Answers (1)

Marius Soutier
Marius Soutier

Reputation: 11274

That setting specifically applies to the history server. If you don't have one started (it's typically used with YARN and Mesos I believe), then the setting you're after is spark.ui.retainedJobs. Check the Spark UI configuration parameters for more details.

These settings only apply to jobs, so in order to pass them to the master itself, check the spark.deploy options in the stand-alone deployment section. You can set them via the SPARK_MASTER_OPTS environment variable.

If you want to clean the data files produced by workers, check the spark.worker.cleanup options in the same section. You can set them via the SPARK_WORKER_OPTS environment variable on your workers.

Upvotes: 1

Related Questions