CODEWITHSUNDEEP
bashdatetimelogfiles
Scott
Scott

Reputation: 6736

Parsing log files in BASH to find "ERROR" entries between certain time stamps

I am writing a script in BASH that needs to check through log files for ERROR entries. I plan to run this as a cron hourly, so I only want to have it only return ERROR type entries that occurred within the last hour (all server times are GMT). I establish the following variables

# Log file directory
LOGPATH="/path/to/logs/"
# Current date and time
CURDATE=`date +%Y-%m-%d`
CURTIME=`date +%H:%M:%S`
# Old date and time
OLDDATE=`date +%Y-%m-%d -d "1 hour ago"`
OLDTIME=`date +%H:%M:%S -d "1 hour ago"`

All log files adhere to the file name format of ktYEAR-MONTH-DAY.root.log.txt Where YEAR/MONTH/DAY are replaced with the date that entries are recorded in. So for instance, today's log file would be kt2011-08-15.root.log.txt. An example entry of the contents is

2011-08-15 | 19:30:02 | ERROR | 18333 | 337 | n/a | dms | default | error | XMLRPC Lucene - addDocument - Reason: Failed to parse XML-RPC request: An invalid XML character (Unicode: 0xb) was found in the element content of the document.

The columns of interest are the 1st, 2nd, 3rd (value may be "INFO", "DEBUG", etc, but am only interested when "ERROR" is the value) and the last column which is the body of the log message.

What I am trying to accomplish is having this BASH script parse through the file(s) that have entries spanning the last hour of activity (as defined in the 1st and 2nd column), and if the 4th column contains the string "ERROR", then display the right-most column's contents. My confusion comes when trying to determine how to parse through the log file(s) based off of the $CURTIME an $OLDTIME, made worse when midnight comes and I then have to search through the previous day's log file. I would prefer not to do a blanket grep style search through all the log files as the quantity and size can be excessive, but if that's how it has to be done, then so be it.

Upvotes: 0

Views: 2288

Answers (2)

jfg956
jfg956

Reputation: 16750

This is as simple as doing string comparison in awk. When you pass midnight, simply add the $OLDDATE file to the search:

if [ "$CURDATE" != "$OLDDATE" ]; then
   cat "kt$OLDDATE.root.log.txt" "kt$CURDATE.root.log.txt"
else
   cat "kt$CURDATE.root.log.txt"
fi | awk -F "|" -v olddate=$OLDDATE -v oldtime=$OLDTIME -v curdate=$CURDATE 'BEGIN{olddate=olddate " "; curdate = curdate " "; oldtime = " " oldtime " "}
$1 == olddate && $2 >= oldtime && $3 == " ERROR "{print $0}
$1 > olddate && $3 == " ERROR "{print $0}'

Can be combined with glenn's solution to be much shorter.

Upvotes: 1

glenn jackman
glenn jackman

Reputation: 246799

 awk -F ' \\| ' -v "d=$(date -d "1 hour ago" -u +%Y-%m-%d@%H:%M:%S)" '$3 == "ERROR" && $1"@"$2 > d'

Upvotes: 2

Related Questions