CODEWITHSUNDEEP
linuxbashawkhuman-readable
Devon Bessemer
Devon Bessemer

Reputation: 35337

Convert human readable to bytes in bash

So I am trying to analyze very large log files in linux and I have seen plenty of solutions for the reverse of this, but the program that records the data doesn't allow for output formatting therefore it only outputs in human readable format (I know, what a pain). So the question is: How can I convert human readable to bytes using something like awk:

So converting this:

937
1.43K
120.3M

to:

937
1464
126143693

I can afford and I expect some rounding errors.

Thanks in advance.

P.S. Doesn't have to be awk as long as it can provide in-line conversions.

I found this but the awk command given doesn't appear to work correctly. It outputs something like 534K"0".

I also found a solution using sed and bc, but because it uses bc it has limited effectiveness meaning it only can use one column at a time and all the data has to be appropriate for bc or else it fails.

sed -e 's/K/\*1024/g' -e 's/M/\*1048576/g' -e 's/G/\*1073741824/g' | bc

Upvotes: 26

Views: 18015

Answers (5)

tink
tink

Reputation: 15214

$ cat dehumanise 
937
1.43K
120.3M

$ awk '/[0-9]$/{print $1;next};/[mM]$/{printf "%u\n", $1*(1024*1024);next};/[kK]$/{printf "%u\n", $1*1024;next}' dehumanise
937
1464
126143692

Upvotes: 8

brablc
brablc

Reputation: 1791

Use numfmt --from=iec from GNU coreutils.

Upvotes: 46

Yzmir Ramirez
Yzmir Ramirez

Reputation: 1281

awk 'function pp(p){printf "%u\n",$0*1024^p} /[0-9]$/{print $0}/K$/{pp(1)}/M$/{pp(2)}/G$/{pp(3)}/T$/{pp(4)}/[^0-9KMGT]$/{print 0}'

This is a modification on @starfry's answer.

Let's break it down:

function pp(p) { printf "%u\n", $0 * 1024^p }

Define a function named pp that takes a single parameter p and prints the $0 multiplied by 1024 raised to the p-th power. The %u will print the unsigned decimal integer of that number.

/[0-9]$/ { print $0 }

Match lines that end with a digit (the $ matches the end of the line), then run the code inside the { and }. Print the entire line ($0)

/K$/ { pp(1) }

Match lines that end with the capital letter K, call the function pp() and pass 1 to it (p == 1). NOTE: When $0 (e.g. "1.43K") is used in a math equation only the beginning numbers (i.e. "1.43") will be used below. Example with $0 = "1.43K"

$0 * 1024^p == 1.43K * 1024^1 == 1.43K * 1024 = 1.43 * 1024 = 1464.32

/M$/ { pp(2) }

Match lines that end with the capital letter M, call the function pp() and pass 2 to it (p == 2). Example with $0 == "120.3M"

$0 * 1024^p == 120.3M * 1024^2 == 120.3M * 1024^2 == 120.3M * 1024*1024 = 120.3 * 1048576 = 126143692.8

etc... for G and T

/[^0-9KMGT]$/ { print 0 }

Lines that do not end with a digit or the capital letters K, M, G, or T print "0".

Example:

$ cat dehumanise
937
1.43K
120.3M
5G
933G
12.2T
bad
<>

Results:

$ awk 'function pp(p){printf "%u\n",$0*1024^p} /[0-9]$/{print $0}/K$/{pp(1)}/M$/{pp(2)}/G$/{pp(3)}/T$/{pp(4)}/[^0-9KMGT]$/{print 0}' dehumanise
937
1464
126143692
5368709120
1001801121792
13414041858867
0
0

Upvotes: 1

starfry
starfry

Reputation: 9953

Here's a function that understands binary and decimal prefixes and is easily extendable for large units should there be a need:

dehumanise() {
  for v in "${@:-$(</dev/stdin)}"
  do  
    echo $v | awk \
      'BEGIN{IGNORECASE = 1}
       function printpower(n,b,p) {printf "%u\n", n*b^p; next}
       /[0-9]$/{print $1;next};
       /K(iB)?$/{printpower($1,  2, 10)};
       /M(iB)?$/{printpower($1,  2, 20)};
       /G(iB)?$/{printpower($1,  2, 30)};
       /T(iB)?$/{printpower($1,  2, 40)};
       /KB$/{    printpower($1, 10,  3)};
       /MB$/{    printpower($1, 10,  6)};
       /GB$/{    printpower($1, 10,  9)};
       /TB$/{    printpower($1, 10, 12)}'
  done
}

example:

$ dehumanise 2K 2k 2KiB 2KB 
2048
2048
2048
2000

$ dehumanise 2G 2g 2GiB 2GB 
2147483648
2147483648
2147483648
2000000000

The suffixes are case-insensitive.

Upvotes: 20

ThorSummoner
ThorSummoner

Reputation: 18109

Python tools exist

$pip install humanfriendly  # Also available as a --user install in ~/.local/bin

$humanfriendly --parse-size="2 KB"
2000
$humanfriendly --parse-size="2 KiB"
2048

Upvotes: 8

Related Questions