CODEWITHSUNDEEP
hadoophdfssequencefile
dewet
dewet

Reputation: 31

How to copy the output of -text HDFS command into another file?

Is there any way we can copy text content of hdfs file into another file system using HDFS command:

 hadoop fs -text /user/dir1/abc.txt

Can I print the output of -text into another file by using -cat or any method ?:

 hadoop fs -cat /user/deepak/dir1/abc.txt

Upvotes: 2

Views: 8956

Answers (4)

Y.Prithvi
Y.Prithvi

Reputation: 1221

you can use following:

  1. copyToLocal
    hadoop dfs -copyToLocal /HDFS/file /user/deepak/dir1/abc.txt
  2. getmerge
    hadoop dfs -getmerge /HDFS/file /user/deepak/dir1/abc.txt
  3. get
    hadoop dfs -get /HDFS/file /user/deepak/dir1/abc.txt

Upvotes: 0

dewet
dewet

Reputation: 31

Thank you I did use streaming jar example in hadoop-home lib folder as follow :

hadoop -jar hadoop-streaming.jar -input hdfs://namenode:port/path/to/sequencefile \
-output /path/to/newfile -mapper "/bin/cat" -reducer "/bin/cat" \
-file "/bin/cat" -file "/bin/cat" \
-inputformat SequenceFileAsTextInputFormat

you can use "/bin/wc" in case you would like to count the number of lines at the hdfs sequence file.

Upvotes: 0

samthebest
samthebest

Reputation: 31515

As a general command line tip you can use | to another program or > or >> to a file, e.g.

# Will output to standard output (console) and the file /my/local/file
# this will overwrite the file, use ... tee -a ... to append
hdfs dfs -text /path/to/file | tee /my/local/file

# Will redirect output to some other command
hdfs dfs -text /path/to/file | some-other-command

# Will overwrite /my/local/file
hdfs dfs -text /path/to/file > /my/local/file

# Will append to /my/local/file
hdfs dfs -text /path/to/file >> /my/local/file

Upvotes: 0

Aleksei Shestakov
Aleksei Shestakov

Reputation: 2538

As it's written in the documentation you can use hadoop fs -cp to copy files in hdfs. You can use hadoop fs -copyToLocal to copy files from hdfs to local file system. If you want to copy files from one hdfs to another then use DistCp tool.

Upvotes: 2

Related Questions