CODEWITHSUNDEEP
sizehdfs
dezzinto
dezzinto

Reputation: 467

HDFS calculate size of subfolders

Please advice on how can I calculate the size of subfolders in HDFS and sort them by size? hdfs dfs -ls -h /mds/snapshots/user/data | du -sh * | sort -rh | head -10 Seems it should work - but as I understand hdfs doesn't work with additional commands after |

Upvotes: 0

Views: 2371

Answers (1)

jedijs
jedijs

Reputation: 563

You can use:

hdfs dfs -du -s /path/* | sort -r -k 1 -g | awk '{ suffix="KMGT"; for(i=0;
$1>1024 && i < length(suffix); i++) $1/=1024; print int($1) substr(suffix, i,
1), $3; }'

Upvotes: 3

Related Questions