CODEWITHSUNDEEP
hadoophdfs
CIAduck
CIAduck

Reputation: 85

How to restore HDFS blocks moved out of /dataN/dfs/dn/current directory?

Due to an unfortunate series of events, a program moved blocks from /dataN/dfs/dn/current/BP-XXXXXXX/current/finalized/subdirN/subdirN/blk_NNNNNNNNNN into /tmp/blk_NNNNNNNNNN

I don't have any logging from the program to tell where the original subdirN/subdirN/ directory was.

Is there any way to figure out where this block should be based on fsimage file, the block file itself, or some other metadata?

I was able to restore some blocks by looking for the corresponding *.meta file, but there are still some holes. Replication saved me from the worst of it, but I'm still missing 5 "mission critical" files I'd like to try and recover.

From hdfs fsck / I can tell what the missing blocks are, and what HDFS files they belonged to, but I can't tell where in the blockpool they should have been placed.

hdfs fsck / -delete is NOT a solution. I don't want to delete things, I want to try my hardest to recover the files, because I HAVE the blocks. I just don't know where they go.

$ hdfs version Hadoop 2.6.0-cdh5.4.4

Upvotes: 1

Views: 427

Answers (2)

CIAduck
CIAduck

Reputation: 85

Here is what I ended up doing to fix this. This wont work in all cases, but it worked in mine.

I took advantage of the fact that the input file separator would be the "line input record separator" and that the blocks in hadoop could be concatenated with the missing block. Order of the data doesn't matter for me, only that all the lines are there.

I simply retrieved all the blocks for the file (including the one no longer in hdfs that was moved to a new location), concatenated them all together. Deleted the file from HDFS and did an hdfs -put of the contaminated file to restore the contents.

Not perfect, but it was effective. This saved me from having to reverse engineer anything, and also proved the easiest way to restore the data.

Thanks for the help. I'm sure there is useful information in here for the next person with this problem.

Upvotes: 0

Serhiy
Serhiy

Reputation: 4141

Not sure if it possible to do the restore manually, but you can try.

The subdirs are calculated in the: DatanodeUtil.idToBlockDir(...) using the following code:

int d1 = (int)((blockId >> 16) & 0xff);
int d2 = (int)((blockId >> 8) & 0xff);
String path = DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP + DataStorage.BLOCK_SUBDIR_PREFIX + d2;

If the files were moved manually, the fsimage might still contain the block ids, use hdfs oiv command to convert fsimage to XML and get the blockIds by deleted file names.

Upvotes: 3

Related Questions