Reputation: 501
I have a sample log file like below sample.txt: These errors are identical but with different timestamps and error code
catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:2017-08-19 15:43:59.063 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514880 Listener={set}
catalina.out:2017-08-19 15:43:59.703 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514896 Listener={set}
catalina.out:2017-08-19 16:19:41.274 [1526172869 tomcat-http--1] [TEST ECHO]: [J] Connection SetExceptionListener conn=4515735 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
is there any way i can grep or do filtering on this and get output with uniq error lines example expected output
catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
tried
grep Example.txt | uniq > out.txt
but didn't work
The log file contents may vary but the idea is grep for uniq error lines and ignore the repetitive lines
Upvotes: 0
Views: 159
Reputation: 204731
Here's a start:
$ awk '{err=$0; gsub(/[0-9]+/,0,err)} !seen[err]++' file
catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:2017-08-19 16:19:41.274 [1526172869 tomcat-http--1] [TEST ECHO]: [J] Connection SetExceptionListener conn=4515735 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
Add more changes to err
to normalize the various parts of the error messages as you see fit, e.g.:
$ awk '{err=$0; gsub(/[0-9]+/,0,err); gsub(/\[[^]]+]/,"[]",err)} !seen[err]++' file
catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
Upvotes: 1