CODEWITHSUNDEEP
hadoophbaseclouderahortonworks-data-platform
etp
etp

Reputation: 3

HBase MasterProcWALs issue

I noticed that due to some ongoing bug, the Hbase MasterProcWALs folder has filled up my Hdfs. I wanted to know if removing the files under the MasterProcWALs folder will remove any of the data in Hbase?

Upvotes: 0

Views: 2376

Answers (2)

Khairul
Khairul

Reputation: 9

We have newly installed CDP-DC 7.1.3 and getting frequently "This health test is bad because the Service Monitor did not find an active Master." After cleaning the MasterProcWALs from HDFS , it is working fine.

Upvotes: 0

Romain
Romain

Reputation: 21888

MasterProcWALs are used by master nodes, here is a description given by the Apache HBase ™ Reference Guide

HMaster records administrative operations and their running states, such as the handling of a crashed server, table creation, and other DDLs, into a Procedure Store. The Procedure Store WALs are stored under the MasterProcWALs directory. The Master WALs are not like RegionServer WALs. Keeping up the Master WAL allows us run a state machine that is resilient across Master failures. For example, if a HMaster was in the middle of creating a table encounters an issue and fails, the next active HMaster can take up where the previous left off and carry the operation to completion. Since hbase-2.0.0, a new AssignmentManager (A.K.A AMv2) was introduced and the HMaster handles region assignment operations, server crash processing, balancing, etc., all via AMv2 persisting all state and transitions into MasterProcWALs rather than up into ZooKeeper, as we do in hbase-1.x. This section pertains to hbase-2.0.0 through hbase-2.2.x.

So MasterProcWALs contain DDLs procedures not applied by the Active HBase Master on the permanent storage. If you remove them and either you restart the HBase Master or switch to another one, it will not be able to read the procedures stored in these WAL and they will be lost. The master will start in the last known state before the WAL. Data are managed at RegionServers level and should not be affected except if their structure is affected by some procedures stored in the WAL and lost.

Before doing that I would

  • Check what is going on in the "Procedures" tab of the HBase UI, it displays the procedures and the WAL files. Try to figure out the root cause of the problem, the number of WAL files should not increase over the time.
  • Try to restart a master, check in its log because it will read the MasterProcWAL files during its startup.
  • Remove them only if there is a blocking issue during HBase Master startup. If you take the decision to remove them perform a backup, do it with care and at your own risk...

See also HBase Master won't start.

Upvotes: 2

Related Questions