CODEWITHSUNDEEP
hbase
Cornelius
Cornelius

Reputation: 3616

Unable to access HBase table

I have a cluster of 5 nodes on which I created a few tables on HBase and populated it with data. One of the nodes failed and now I am able to list all of the tables on HBase but a scan or disable of two of the tables gives

NativeException: org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in .META. for region TableName,,1295871604968

I want to drop this table but cannot get past this problem. I'm using the hbase shell for this.

Upvotes: 3

Views: 10596

Answers (5)

lmsurprenant
lmsurprenant

Reputation: 1935

In our case, we were seeing this error when someone accidentally left one of our tables in a 'disabled' state. This resulted in the table having 0 regions and therefor "No server address listed in hbase:meta". The fix was to realize it was disabled and then re-enable it:

hbase is_disabled 'table name'

hbase enable 'table name'

Upvotes: 0

prashant
prashant

Reputation: 1

try

flush '.META.'

and

major_compact '.META.'

in the hbase shell & try scanning the table again.

Upvotes: 0

WattsInABox
WattsInABox

Reputation: 4636

We ran into a similar issue, but we swear we didn't have a node go down. However, regardless of how it happened (since we're still not quite sure), how we fixed it was we took a snapshot of the messed-up table and recreated the table from that snapshot.

scan 'mytable'
ERROR: No server address listed in hbase:meta for region mytable,,1408136497251.5110a0bae8315ed52af93663401ab415. containing row

Turns out even though the table is messed up in the meta table, the data was not messed up and we were able to take a new snapshot of it, drop, and clone.

snapshot 'mytable', 'mytable-Snapshot-2014-08-25'
disable 'mytable'
drop 'mytable'
clone_snapshot 'mytable-Snapshot-2014-08-25', 'mytable'

Upvotes: 0

Cornelius
Cornelius

Reputation: 3616

I resolved the problem by deleting the table entries in the .META. table for the tables that gave the error and then recreated the table. I think the .META. table saved the location to the region on the dead node and where unable to update the region location for some reason when the node died. After the entries was deleted I was able to recreate the table. I found a discussion about it here that help send me on the right path.

Upvotes: 2

David
David

Reputation: 3261

Is it possible to restart the node?

HBase should have redistributed the regions to other nodes. There are probably a few reasons that this wouldn't have happened, but the one I can think of off the top of my head is that you have a corrupted HDFS. This could happen if you don't have replication in your data nodes (or replication turned below the number of number of node failures). Check your file system (hadoop fsck /)

Also, it is helpful to list the version of hbase and hadoop you are using. If possible, move to HBase version 0.90 with the hadoop 0.20-append branch (included in CDH3)

Upvotes: 1

Related Questions