Hbase 0.92.1: how to fix .META. table ，because it lost some regioninfo

Question

My Hbase version is 0.92.1.

I found my reduce job have exception like :

java.io.IOException: HRegionInfo was null or empty in .META., row=keyvalues={single,20150411 hxmyxy2013 REF// 336d5ebc5436534e61d16e63ddfca327 7f92f92e8e693531c12a1dcc773f9ffa,1430298962344.ea72df9a3703e476b9f7e3368be47aa2./info:server/1430356486086/Put/vlen=28, single,20150411 hxmyxy2013 REF// 336d5ebc5436534e61d16e63ddfca327 7f92f92e8e693531c12a1dcc773f9ffa,1430298962344.ea72df9a3703e476b9f7e3368be47aa2./info:serverstartcode/1430356486086/Put/vlen=8}

Then i ran command: hbase hbck

which output 57 errors, like:

ERROR: Region hdfs://hadoop01.yupoo-inc.com:9000/hbase/single/fa921b223692d89be274979e1e8f1674 on HDFS, but not listed in META or deployed on any region server

=====

Summary:

-ROOT- is okay.

Number of regions: 1

Deployed on:  hadoop02.yupoo-inc.com,60020,1430643409216

.META. is okay.

Number of regions: 1

Deployed on:  hadoop01.yupoo-inc.com,60020,1430643412565

global is okay.

Number of regions: 1

Deployed on:  hadoop03.yupoo-inc.com,60020,1430643409040

single is okay.

Number of regions: 2081

Deployed on:  hadoop01.yupoo-inc.com,60020,1430643412565 hadoop02.yupoo-inc.com,60020,1430643409216 hadoop03.yupoo-inc.com,60020,1430643409040 hadoop04.yupoo-inc.com,60020,1430643408453 hadoop05.yupoo-inc.com,60020,1430643407996

summary is okay.

Number of regions: 2540

Deployed on:  hadoop01.yupoo-inc.com,60020,1430643412565 hadoop02.yupoo-inc.com,60020,1430643409216 hadoop03.yupoo-inc.com,60020,1430643409040 hadoop04.yupoo-inc.com,60020,1430643408453 hadoop05.yupoo-inc.com,60020,1430643407996

total is okay.

Number of regions: 172

Deployed on:  hadoop01.yupoo-inc.com,60020,1430643412565 hadoop02.yupoo-inc.com,60020,1430643409216 hadoop03.yupoo-inc.com,60020,1430643409040 hadoop04.yupoo-inc.com,60020,1430643408453 hadoop05.yupoo-inc.com,60020,1430643407996

urls is okay.

Number of regions: 5

Deployed on:  hadoop02.yupoo-inc.com,60020,1430643409216 hadoop03.yupoo-inc.com,60020,1430643409040

57 inconsistencies detected.

Status: INCONSISTENT

=====

It seems like some regioninfo were not in meta table, So i asked google and found command like hbase hbck -repair or -fixMeta -fixAssignments or -repairHoles but my hbase version do not have this command ,i run command hbase hbck --help ,it output:

Unknown command line option : --help

Usage: fsck [opts]

where [opts] are:

-details Display full report of all regions.

-timelag {timeInSeconds} Process only regions that have not experienced any metadata updates in the last {{timeInSeconds} seconds.

-fix Try to fix some of the errors.

-sleepBeforeRerun {timeInSeconds} Sleep this many seconds before checking if the fix worked if run with -fix

-summary Print only summary of the tables and status.

-metaonly Only check the state of ROOT and META tables.

but when i run hbase hbck -fix it do nothing , so can anyone help me with this?

Answer 1

I fix this problem by myself.

I found my reduce task failed with this exception:

java.io.IOException: HRegionInfo was null or empty in .META., row=keyvalues={single,20150411 hxmyxy2013 REF// 336d5ebc5436534e61d16e63ddfca327 7f92f92e8e693531c12a1dcc773f9ffa,1430298962344.ea72df9a3703e476b9f7e3368be47aa2./info:server/1430356486086/Put/vlen=28, single,20150411 hxmyxy2013 REF// 336d5ebc5436534e61d16e63ddfca327 7f92f92e8e693531c12a1dcc773f9ffa,1430298962344.ea72df9a3703e476b9f7e3368be47aa2./info:serverstartcode/1430356486086/Put/vlen=8}

Then i check in the Hbase shell by running this command: get '.META.', 'single,20150411 hxmyxy2013 REF// 336d5ebc5436534e61d16e63ddfca327 7f92f92e8e693531c12a1dcc773f9ffa,1430298962344.ea72df9a3703e476b9f7e3368be47aa2.'

Which output is :

COLUMN CELL

info:server timestamp=1430356486086, value=hadoop05.yupoo-inc.com:60020

info:serverstartcode timestamp=1430356486086, value=1430352791855

2 row(s) in 0.0080 seconds

Then I delete this data which key is :

single,20150411 hxmyxy2013 REF// 336d5ebc5436534e61d16e63ddfca327 7f92f92e8e693531c12a1dcc773f9ffa,1430298962344.ea72df9a3703e476b9f7e3368be47aa2.