CODEWITHSUNDEEP
linuxfilesizedu
slesh
slesh

Reputation: 2007

Could get right file size using du

I see that my bucket size on the aws s3 storage is 13.2GiB and it has 1570 files:

$ aws s3 ls --summarize --human-readable s3://mybucket/ | grep -E "(Total\sObjects|Total\sSize)"
Total Objects: 1570
   Total Size: 13.2 GiB

When I downloaded this bucket here is what I see:

$du -sh ./test
14G
$wc -l ./test
1570
$ du -sb ./test
14204477032
$ du -sb ./test | awk '{ \
            split( "B KB MB GB" , v ); \
            s=1; \
            while( $1>=1024 ) { \
                $1/=1024; s++ \
            } \
            printf "%.1f%s", $1, v[s] \
        }'
13.2GB

How to achieve the same result using standard Linux functions?

Upvotes: 0

Views: 1247

Answers (1)

Alfe
Alfe

Reputation: 59416

du is originally for finding out how much space a file occupies on the storage medium (disk). That's the main reason why it rather rounds up than down. A started allocated block is always "used" completely, even if just two bytes of it are in use.

Your case rather seems to aim at counting the bytes in the files, regardless of the storage space they occupy. For this, du has the option --apparent-size. Rather than disk usage, it then displays the file's sizes. Combined with --block-size=1 this is simpler spelled as -b.

Next thing is that you want to convert a large number like 14204477032 into a neat version like 13.2GB. You also state in a comment that 14G (as -h would produce) isn't precise enough for your taste, and you also provide an awk script which does exactly this conversion so that you already have a working solution.

I'm not aware of any standard Unix tool other than awk or even more complex things like perl or python which would do this in a much easier fashion. There are other people looking for a solution for this, and yours is among the best ones.

SO my advice is just this: Stick with your solution. The only improvement I'd propose would be to use bit-shifting (>> 10) instead of division (/ 1024) but that's rather a matter of taste.

Upvotes: 1

Related Questions