CODEWITHSUNDEEP
perlbashshellawk
Adam Siemion
Adam Siemion

Reputation: 16039

Parse log file and print the number of log entries per second

Log file:

2014-05-29 07:37:57 [Thread-8] TRACE ClassImpl - logging message
2014-05-29 07:37:57 [Thread-7] TRACE ClassImpl - logging message
2014-05-29 07:37:58 [Thread-3] TRACE ClassImpl - logging message
2014-05-29 07:37:58 [Thread-5] TRACE ClassImpl - logging message
2014-05-29 07:37:58 [Thread-8] TRACE ClassImpl - logging message
2014-05-29 07:37:59 [Thread-7] TRACE ClassImpl - logging message

I would like to print the number of log entries per second, for the above log file that would be:

2014-05-29 07:37:57 = 2
2014-05-29 07:37:58 = 3 
2014-05-29 07:37:59 = 1

My simplistic approach using bash is too slow:

for h in $(seq 0 7); do 
    for m in $(seq 1 60); do 
        for s in $(seq 1 60); do 
            echo -n "$h:$m:$s="; grep "$h:$m:$s" server.log|wc -l; 
        done; 
    done; 
done

You can ignore the date as I am mostly interested in time.

awk solution by Vijay:

time awk '{a[$1" "$2]++}END{for(i in a){print i" = "a[i]}}' server.log > /dev/null

real    0m0.475s
user    0m0.355s
sys 0m0.096s

Perl solution by mpapec:

time perl -anE'$h{$_}++ or push @r,$_ for "@F[0,1]" }{say "$_ = $h{$_}" for@r' server.log > /dev/null

real    0m4.561s
user    0m4.235s
sys 0m0.120s

Upvotes: 1

Views: 246

Answers (3)

mpapec
mpapec

Reputation: 50647

perl -anE'$h{$_}++ or push @r,$_ for "@F[0,1]" }{say "$_ = $h{$_}" for@r' file

output

2014-05-29 07:37:57 = 2
2014-05-29 07:37:58 = 3
2014-05-29 07:37:59 = 1

faster version

perl -nE'$h{$}++ or push @r,$ for /(\S+\s+\S+)/ }{say "$_ = $h{$_}" for@r' file

perl -nE'$h{$_}++ or push @r,$_ for substr($_,0,19)}{say "$_ = $h{$_}" for@r' file

Upvotes: 1

sat
sat

Reputation: 14949

Another way:

cut -d' ' -f1,2 yourfile.log | uniq -c | awk '{ print $2,$3,"=",$1}'

Upvotes: 0

Vijay
Vijay

Reputation: 67231

awk:

awk '{a[$1" "$2]++}END{for(i in a){print i" = "a[i]}}' your_file

Perl:

perl -lane '$x{$F[0]." ".$F[1]}++;
            END{print $_." = ".$x{$_} for(keys(%x))}' your_file

Upvotes: 2

Related Questions