CODEWITHSUNDEEP
bashawk
Danielo515
Danielo515

Reputation: 7051

AWK performance while processing big files

I have an awk script that I use for calculate how much time some transactions takes to complete. The script gets the unique ID of each transaction and stores the minimum and maximum timestamp of each one. Then it calculates the difference and at the end it shows those results that are over 60 seconds.

It works very well when used with some thousand (200k) but it takes more time when used in real world. I tested it several times and it takes about 15 minutes to process about 28 million of lines. Can I consider this good performance or it is possible to improve it?

I'm open to any kind of suggestion.

Here you have the complete code

zgrep -E "\(([a-z0-9]){15,}:" /path/to/very/big/log |  awk '{
gsub("[()]|:.*","",$4); #just removing ugly chars
++cont
min=$4"min" #name for maximun value of current transaction
max=$4"max" #same as previous, just for readability 
split($2,secs,/[:,]/) #split hours,minutes and seconds
seconds = 3600*secs[1] + 60*secs[2] + secs[3] #turn everything into seconds
if(arr[min] > seconds || arr[min] == 0)
  arr[min]=seconds
if(arr[max] < seconds)
   arr[max]=seconds
dif=arr[max] - arr[min]
if(dif > 60)
  result[$4] = dif
}
END{
for(x in result)
   print x" - "result[x]
print ":Processed "cont" lines"
}'

Upvotes: 1

Views: 1350

Answers (2)

Danielo515
Danielo515

Reputation: 7051

After making some test, and with the suggestions gave by Ed Morton (both for code improvement and performance test) I found that the bottleneck was the zgrep command. Here is an example that does several things:

  • Check if we have a transaction line (first if)
  • Cleans the transaction id
  • checks if this has been already registered (second if) by checking if it is in the array
  • If is not registered then checks if it is the appropriate type of transaction and if so it registers the timestamp in second
  • If is already registered saves the new time-stamp as the maximun
  • After all it makes the necessary operations to calculate the time difference

Thank you very much to all that helped me.

zcat /veryBigLog.gz |  awk '
{if($4 ~ /^\([:alnum:]/ ){
    name=$4;gsub(/[()]|:.*/,"",name);
    if(!(name in min)){
        if($0 ~ /TypeOFTransaction/ ){
          split($2,secs,/[:,]/) 
          seconds = 3600*secs[1] + 60*secs[2] + secs[3] 
          max[name] = min[name]=seconds
          print lengt(min) "new "name " start at "seconds
       }
       }else{ 
            split($2,secs,/[:,]/)
            seconds = 3600*secs[1] + 60*secs[2] + secs[3]
            if( max[name] < seconds) max[name]=seconds
            print name " new max " max[name]
        }
        }}END{
            for(x in min){
                dif=max[x]- min[x]
                print max[x]" max - min "min[x]" : "dif
            }
            print "Processed "NR" Records"
            print "Found "length(min)" MOs" }'

Upvotes: 2

Ed Morton
Ed Morton

Reputation: 203229

You don't need to calculate the dif every time you read a record. Just do it once in the END section.

You don't need that cont variable, just use NR.

You dont need to populate min and max separately string concatenation is slow in awk.

You shouldn't change $4 as that will force the record to be recompiled.

Try this:

awk '{
    name = $4
    gsub(/[()]|:.*/,"",name); #just removing ugly chars

    split($2,secs,/[:,]/) #split hours,minutes and seconds
    seconds = 3600*secs[1] + 60*secs[2] + secs[3] #turn everything into seconds

    if (NR==1) {
        min[name] = max[name] = seconds
    }
    else {
        if (min[name] > seconds) {
            min[name] = seconds
        }
        if (max[name] < seconds) {
            max[name] = seconds
        }
    }
}

END {
    for (name in min) {
        diff = max[name] - min[name]
        if (diff > 60) {
            print name, "-", diff
        }
    }
    print ":Processed", NR, "lines"
}'

Upvotes: 6

Related Questions