CODEWITHSUNDEEP
hadoopgethdfs
Giovanni Tusa
Giovanni Tusa

Reputation: 11

How to copy a data directory from hdfs to local fs?

I started with the need to backup the whole hadoop datanode data directory using:

hdfs dfs -copyToLocal /var/hadoop/dfs/name/data /home/ubuntu/hadoopfiles

And I got an error:

"No such file opr directory" for /var/hadoop/dfs/name/data

After some search I've found this Stack Overflow topic with examples: https://stackoverflow.com/questions/28213116/hadoop-copy-a-local-file-system-folder-to-hdfs#=

But even when I do:

hdfs dfs -ls

I receive the error message ls: `.': No such file or directory

I have also looked to other posts, it seems this is a quite common issue, but I was unable to find a solution for me.

Thanks.

Upvotes: 1

Views: 3751

Answers (1)

Arun Solomon
Arun Solomon

Reputation: 421

First, use 

hadoop fs -get /theFolder

 to copy it into the current directory you are ssh'ed into on your box.

Then you can use either scp or my preference of rsync to copy the files between your box and your local system like so. Here's how I'd use rsync after having used the -get, still in the same directory:

rsync -av ./theFolder username@yourlocalmachine:/home/username

This will copy theFolder from the local fs on your box into your home folder on your machine's fs. Be sure to replace username with your actual username in both cases, and yourlocalmachine with your machine's hostname or ip address.

Pls refer this answer

Copy from Hadoop to local machine

Or look this link scp method

Upvotes: 2

Related Questions