CODEWITHSUNDEEP
javafilesystemshadoophdfs
lucky_start_izumi
lucky_start_izumi

Reputation: 2591

About hadoop hdfs filesystem rename

I am storing lots of data into hdfs. And I need to move the files from one folder to another.

May I ask generally how much is the cost of filesystem's rename method? Say I have to move terabytes of data.

Thank you very much.

Upvotes: 4

Views: 5889

Answers (2)

Praveen Sripati
Praveen Sripati

Reputation: 33545

Moving files in HDFS or any file system if implemented properly involves changes to the name space and not moving of the actual data. Going through the code only changes in the name space (memory and edit log) in the Name node are done.

From the NameNode.java class

  • The NameNode controls two critical tables:
  • 1) filename->blocksequence (namespace)
  • 2) block->machinelist ("inodes")

Only the first part needs to be modified, block to machine list need not be. I haven't tried it out, but I guess it should be OK.

Upvotes: 4

dmeister
dmeister

Reputation: 35644

Rename is a metadata-only operation in HDFS. Therefore it is be very cheap like it is in a normal POSIX filesystem, too. No data is moved. The only server involved is the namenode.

The source code for the rename can be found here. Pretty straight forward.

Upvotes: 5

Related Questions