CODEWITHSUNDEEP
shellawksedgrepuniq
Rczone
Rczone

Reputation: 501

Filter out uniq lines from log file

I have a sample log file like below sample.txt: These errors are identical but with different timestamps and error code

catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:2017-08-19 15:43:59.063 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514880 Listener={set}
catalina.out:2017-08-19 15:43:59.703 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514896 Listener={set}
catalina.out:2017-08-19 16:19:41.274 [1526172869 tomcat-http--1] [TEST ECHO]: [J] Connection SetExceptionListener conn=4515735 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)

is there any way i can grep or do filtering on this and get output with uniq error lines example expected output 

catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)

tried 

grep  Example.txt | uniq > out.txt

but didn't work

The log file contents may vary but the idea is grep for uniq error lines and ignore the repetitive lines

Upvotes: 0

Views: 159

Answers (1)

Ed Morton
Ed Morton

Reputation: 204731

Here's a start:

$ awk '{err=$0; gsub(/[0-9]+/,0,err)} !seen[err]++' file
catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:2017-08-19 16:19:41.274 [1526172869 tomcat-http--1] [TEST ECHO]: [J] Connection SetExceptionListener conn=4515735 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)

Add more changes to err to normalize the various parts of the error messages as you see fit, e.g.:

$ awk '{err=$0; gsub(/[0-9]+/,0,err); gsub(/\[[^]]+]/,"[]",err)} !seen[err]++' file
catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)

Upvotes: 1

Related Questions