CODEWITHSUNDEEP
hadoophadoop-yarn
核心力量
核心力量

Reputation: 99

How to clean application history in hadoop yarn?

After applications finished on yarn, a great amount application history will be shown in yarn.

Both in web-ui http://localhost:8088/cluster and command line yarn application -list -appStates FINISHED

How can I clean these history? Or where does yarn store these history information?

Restart RM,NM does not clean these history.

Delete the logs in local file system ${YARN_LOG_DIR}/userlogs does not clean these history.

Delete the logs' directory/files in hdfs:///tmp/logs/username/logs does not clean these history.

Hadoop version: 2.7.7

Upvotes: 2

Views: 3320

Answers (2)

tk421
tk421

Reputation: 5957

If you've enabled log-aggregation, you can set yarn.log-aggregation.retain-seconds to a reasonable value (like a day or a week depending on how many jobs you run) to have YARN purge jobs on a continual basis.

Otherwise set yarn.nodemanager.log.retain-seconds to determine how long you want to keep logs.

You can also disable persistence on restarts by setting yarn.resourcemanager.recovery.enabled to false in your yarn-site.xml.

Refer to yarn-default.xml for all the log configurables in YARN and ResourceManager Restart for more information on persistence.

Upvotes: 3

核心力量
核心力量

Reputation: 99

Oh, I've found the solution.

yarn resourcemanager -format-state-store

and restart the cluster.

But still don't know where yarn store these history, in ZK?

Upvotes: -1

Related Questions