CODEWITHSUNDEEP
bashhadoophdfs
Itération 122442
Itération 122442

Reputation: 2972

How to list files with metadata on one line in HDFS with bash?

I am writing a bash script that scans HDFS and do stuff with the output.

Fetching file names is easy with the following:

    for line in $(hdfs dfs -ls -t -r -C -R $HDFS_CLEANING_STG); do
        echo $line
    done

Output:

/dir
/dir/file1
/dir/file2

However, it removes the file size, date, rights, etc.

With the same method but without the -C flag, it gives the metadata but not on one line only:

Output example:

-rw-rw-r--+
3
hdfs
hdfs
34448169
2020-05-04
11:36
/dir/file

I would like to get these information, but with this output (like a "normal" ls):

-rw-rw-r--+ 3 hdfs hdfs 34448169 2020-05-04 11:36 /dir/file

How can I achieve that ?

Upvotes: 0

Views: 426

Answers (1)

Francesco
Francesco

Reputation: 977

The problem is in the for loop. When you do for line in $(hdfs dfs -ls -t -r -C -R $HDFS_CLEANING_STG); , it iterates over each word, not over each line. So, every time it faces a white space, the value of line changes with the word you just read.

A simple way to fix it is doing like this:

for line in "$(hdfs dfs -ls -t -r -C -R $HDFS_CLEANING_STG)"; do
    echo "$line"
done

Or like this:

hdfs dfs -ls -t -r -C -R $HDFS_CLEANING_STG | while read line; do
    echo "$line"
done

Upvotes: 2

Related Questions