CODEWITHSUNDEEP
hadoophivemapr
knowone
knowone

Reputation: 840

hive tables exist but couldn't find the directories

I have a MapR cluster of 10 nodes which recently undergo some restructuring. Basically there were some missing disks on some nodes that were fixed. After this the hive directories couldn't be found on the HDFS path. However, the hive tables and data are intact and the tables could be seen from hive CLI.

I tried fsck which ran without any issues, then had gfsck executed.
Below is the output of gfsck:

$ sudo /opt/mapr/bin/gfsck -r cluster=tmcluster1 rwvolume=users
Starting GlobalFsck:
  clear-mode        = false
  debug-mode        = false
  dbcheck-mode      = false
  repair-mode       = true
  assume-yes-mode   = false
  cluster       = tmcluster1
  rw-volume-name    = users
  snapshot-name     = null
  snapshot-id       = 0
  user-id       = 0
  group-id      = 0

  get volume properties ...

  put volume users in global-fsck mode ...

  get snapshot list for volume users ...

  starting phase one (get containers) for volume users(31225146) ...
    got volume containers map
  done phase one

  starting phase two (get inodes) for volume users(31225146) ...
java.lang.Exception: ContainerGetInodes RPC retry 11
    at com.mapr.fs.globalfsck.PhaseTwo$PhaseTwoThread.getInodesOfContainer(PhaseTwo.java:238)
    at com.mapr.fs.globalfsck.PhaseTwo$PhaseTwoThread.run(PhaseTwo.java:154)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

In hive, I could only see the tables, structure, etc. I couldn't fire any other DDL. For example a normal count on a table gives following error:

hive> select count(1) from table1;
2015-08-20 21:10:13,4261 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:1306 Thread: 27514 mkdirs failed for /user/mapr/tmp/hive/hive_2015-08-20_21-10-13_254_31142713538709, error 11
FAILED: RuntimeException java.io.IOException: Error: Resource temporarily unavailable(11), file: hive_2015-08-20_21-10-13_254_3114271353870903660-1, user name: mapr, ID: 5000

There's only one user only, 'mapr', for performing all the operations on hive tables.

Anybody got any ideas on this??

Thanks in advance.

Upvotes: 0

Views: 694

Answers (1)

Deb
Deb

Reputation: 471

the CLI is replicating the Table Details coz, the MetaStore DB is containing all the Table Details, along with names Location , Check the Tables in you MySQL/Posrtgress/Derby/Orable, with Hive Meta store DB, Table Name like TBLS, there you will see all the Meta Store Info.

in the Above case, as you have mention you are not able to list all the direcoties it could be possible that the Data Blocks are corrupted {which You have already mention}, so its possible if you are having replecatino, you can call back the Data, but its posibility is very less, some how you can try restarting the metastore service by 

"hive --service metastore" or "sudo service hive-metastore start" or "hive --service metastore"

If the Data is having replication , then Namenode will be try to fix the same, else you have to recreate the tables, again to reupload the Data.

Let Me Know If You haveing any other Option, so that same I can use for mine case. :)

Upvotes: 1

Related Questions