CODEWITHSUNDEEP
bashhadoophdfs
Vektor88
Vektor88

Reputation: 4930

Count number of files with given extension on HDFS folder

I'm writing a bash script that should be able to count the number of json files within a specified folder.

What I'm doing right now is this: 

hdfs dfs -ls /path/to/files/*.json | grep -E '^-' | wc -l

When there is at least one file, it returns the number of results, but when there is no json file, I'd expect the result to be 0, since the path exists but it doesn't contain any files matching the *.json pattern. What I'm getting however, is an error:

ls: `/path/to/files/*.json': No such file or directory

Is this the expected behavior?

Upvotes: 3

Views: 4485

Answers (1)

Deepan Ram
Deepan Ram

Reputation: 850

Yeah , that is the default behaviour. Since these are error messages, they are sent to stderr not stdout.

In case you need to suppress the messages, you can use it like :-

hdfs dfs -ls  <hdfs_location>/*.json -R 2>/dev/null | grep -E '^-' | wc -l

(It would show only 0 (with no error) if no files with the matched extension is found ).

You can also refer : - https://unix.stackexchange.com/questions/82698/how-to-mute-ls-command

Upvotes: 2

Related Questions