CODEWITHSUNDEEP
linuxhadoophdfs
user8587005
user8587005

Reputation: 65

Listing files in hdfs and appending the output to a text file

I want to list the files in hadoop based on a condition and append the output to a text file. Below is the commands which I am using 

hadoop fs -ls /path/ | grep 'xxx' | awk '{print $8}' >> /hdfs_path/test.txt

when I execute the above command it says the below error

-bash: /hdfs_path/test.txt: No such file or directory

but the file exists in the directory.

Upvotes: 0

Views: 3799

Answers (1)

Brandon McKenzie
Brandon McKenzie

Reputation: 1685

The problem is that stdout redirection is only aware of your local filesystem, and knows nothing about Hadoop and cannot natively write to it. You need to stick with the hadoop fs or newer hdfs dfs commands for hdfs interactions.

Try appendToFile as your piped command.

Usage: hdfs dfs -ls /your/hdfs/path | grep 'xxx' | hdfs dfs -appendToFile - hdfs://your.hdfs.system/your/hdfs/path/paths.txt

The only other alternative I have is to save the output to your local filesystem and upload that to HDFS after you're done if you need it there.

Upvotes: 1

Related Questions