CODEWITHSUNDEEP
hadoophadoop2hortonworks-data-platform
Pranjal Mittal
Pranjal Mittal

Reputation: 11422

Shell command to transfer files from HDFS to local filesystem in Hadoop 2.6.9

I am new to Hadoop. Is there a bash command to transfer files from the Hadoop distributed file system to the standard file system on a hadoop node.

I am using Hadoop 2.6.0

I saw another similar question which asks how to do the same in Java: Copying files from HDFS to local file system with JAVA

Can we do it with a simple shell command instead (which runs on a node that is part of the hadoop cluster)?

Upvotes: 2

Views: 2836

Answers (2)

Sravan K Reddy
Sravan K Reddy

Reputation: 1082

If you want to pull data down from HDFS to a local directory,you'll need to use the -get or -copyToLocal switches to the hadoop fs command.

hadoop fs -copyToLocal hdfs://path localpath

just call the command in shell scripting.you can do something like below.

for line in awk '/.csv/ {print $2}' /user/hadoop/TempFiles/CLNewFiles.txt;

do

hadoop fs copyToLocal /user/hadoop/TempFiles/$line yourlocalpath

echo "$line file is downloading from hadoop"

done

Upvotes: 0

Pradeep Gollakota
Pradeep Gollakota

Reputation: 2181

hdfs dfs -get /hdfs/path /local/path

hdfs dfs -put /local/path /hdfs/path

Upvotes: 2

Related Questions