Log file data extraction into csv file

Question

I have a application.log file it has the below data and other data as well. Totally this log file contain 7000 entries. From that i am trying to extract the Params : {"file_list": and request : {"method":"registerSender" datas only.

2020-04-14 13:34:08 INFO  app-proxy:53 - Params : {"file_list":["index.html"],"conn_id":"35d20d60","Token":"m.usha","proxy_name":"India","hash":"03870bc9df9"}

2020-04-14 06:09:57 INFO  app-proxy:53 - Params : {"file_list":["DU_SAMPLE_TERT_BD1234.7z"],"conn_id":"5e8e2cb9","Token":"m.usha","proxy_name":"India","hash":"2a357570"}

2020-04-14 10:32:04 INFO  app-proxy:53 - Params : {"file_list":["DU_SAMPLE_188472.7z"],"conn_id":"42474096","Token":"geetha.priya","proxy_name":"India","hash":"2a357570b"}

2020-04-14 07:34:46 INFO  app-proxy:151 - request : {"method":"registerSender","id":"reqFromProxy","jsonrpc":"2.0","params":{"origin":"India","hashName":"251e9a9742","Token":"nithya.d","fileList":["ATT_A515FXXU2ATD1.tar.md5"]}}

2020-04-14 08:48:02 INFO  app-proxy:53 - Params : {"file_list":["ATT_A515FXXU2ATD1.tar.md5"],"conn_id":"f10eec35","Token":"kumar.mari","proxy_name":"India","hash":"251e9a974222e8ba369d6b6c45628017"}

2020-04-14 01:24:53 INFO  app-proxy:151 - request : {"method":"registerSender","id":"reqFromProxy","jsonrpc":"2.0","params":{"origin":"India","hashName":"e061202","Token":"rajesh.raj","fileList":["OLLAY_HUM.tar.md5"]}}

I tried with below script,

#!/bin/bash

file="application.log"
outputfile="output.log"
if [ -f "$outputfile" ]; then
echo "File exists & deleteing"
rm $outputfile
else
echo "File does not exist"
fi

while read line
do

if echo "$line" | grep 'file_list'; then
   echo $line >> "$outputfile"
elif echo "$line" | grep '"method":"registerSender"'; then
    echo $line >> "$outputfile"
else
    echo "no match";
fi
done < $file

But i could able to filter the file_list & "method":"registerSender" entries from application.log file, Further i am trying to extract the above log data in below diagram format. Any help is appreciated.

Answer 1

Could you please try following.

awk '
BEGIN{
  print "S.No Date Time file_list Token proxy_name"
}
!/file_list|fileList|"method":"registerSender"/{ next }
match($0,/"file_list":\["[^"]*|"fileList":\["[^"]*/){
  fileVal=substr($0,RSTART,RLENGTH)
  gsub(/"file_list":\["|"fileList":\["/,"",fileVal)
}
match($0,/"Token":"[^"]*/){
  token_Val=substr($0,RSTART+9,RLENGTH-9)
}
match($0,/"proxy_name":"[^"]*|"origin":"[^"]*/){
  proxyName=substr($0,RSTART,RLENGTH)
  gsub(/"proxy_name":"|"origin":"/,"",proxyName)
}
{
  split($1,arrayDate,"-")
  print ++count,arrayDate[1]"-"arrayDate[2]"-"arrayDate[3],$2,fileVal,token_Val,proxyName
  fileVal=token_Val=proxyName=""
}
'  Input_file | column -t

Explanation: Detailed explanation added.

awk '                                                        ##Starting awk program from here.
BEGIN{                                                       ##Starting BEGIN section from here.
  print "S.No Date Time file_list Token proxy_name"          ##Printing headers here.
}
!/file_list|fileList|"method":"registerSender"/{ next }      ##If a line not having mentioned strings by OP simply move next then.
match($0,/"file_list":\["[^"]*|"fileList":\["[^"]*/){        ##Using match function to match file_list":\[" till next " to get its value.
  fileVal=substr($0,RSTART,RLENGTH)                          ##saving matched value into variable fileVal here.
  gsub(/"file_list":\["|"fileList":\["/,"",fileVal)          ##substituting "file_list":\[" OR "fileList":\[" with NULL.
}
match($0,/"Token":"[^"]*/){                                  ##Using match function to match "Token":"[ till next ".
  token_Val=substr($0,RSTART+9,RLENGTH-9)                    ##saving matched value into variable tolen_Val here.
}
match($0,/"proxy_name":"[^"]*|"origin":"[^"]*/){             ##using proxy_name":" till " OR origin.
  proxyName=substr($0,RSTART,RLENGTH)                        ##saving matched value into variable proxyName here.
  gsub(/"proxy_name":"|"origin":"/,"",proxyName)             ##substituting "proxy_name":" OR "origin":"with NULL.
}
{
  split($1,arrayDate,"-")                                    ##Splitting 1st column into array with delimiter space here.
  print ++count,arrayDate[1]"-"arrayDate[2]"-"arrayDate[3],$2,fileVal,token_Val,proxyName    ##Printing all the values to get OP result here.
  fileVal=token_Val=proxyName=""                             ##Nullifying variables here.
}
'  Input_file | column -t                                    ##Mentioning Input_file name here and passing its output to column command.