CODEWITHSUNDEEP
logstashlogstash-grok
aditya soni
aditya soni

Reputation: 65

Custom Grok Pattern for logs

So here is a sample of my log:

23:28:32.226 WARN  [MsgParser:ListProc-Q0:I5]   Parsing error
Error mapping the fieldAdditional Information: 

    at com.authentic.mapper.parsing.LengthVar.readBytes(LengthVar.java:178)
    at com.authentic.mapper.parsing.GrpLengthVar.read(GrpLengthVar.java:96)
    at com.authentic.mapper.parsing.GrpLengthVar.read(GrpLengthVar.java:119)
    at com.authentic.mapper.parsing.MsgParser.processReadEnumeration(MsgParser.java:339)
    at com.authentic.mapper.parsing.MsgParser.parseIncomingMessageBody(MsgParser.java:295)
    at com.authentic.mapper.MapperMgr.parseMsg(MapperMgr.java:1033)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.parseMessage(AbstractConnectionHandler.java:4408)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.plainMessageReceivedEvent(AbstractConnectionHandler.java:2031)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.messageReceivedEvent(AbstractConnectionHandler.java:1911)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:801)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:282)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:261)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.processEventQueue(AbstractConnectionHandler.java:4110)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.access$100(AbstractConnectionHandler.java:320)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler$ConnectionHandlerRunner.execute(AbstractConnectionHandler.java:416)
    at com.authentic.architecture.actions.ListProcessor.suspend(ListProcessor.java:1130)
    at com.authentic.architecture.actions.ListProcessor.run(ListProcessor.java:775)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: "^123"
    at java.lang.NumberFormatException.forInputString(Unknown Source)
    at java.lang.Integer.parseInt(Unknown Source)
    at java.lang.Integer.parseInt(Unknown Source)
    at com.authentic.mapper.parsing.LengthVar.readBytes(LengthVar.java:170)
    ... 17 more

I have to parse this logs into following fields: timestamp, log-level, logger, msg, stacktrace.

i have used the multiline filter:

multiline {
pattern => "%{TIME:timestamp}"
negate => true
what => “previous”
}

and the pattern i used in grok filter:

match=>{"message"=>"%{TIME:timestamp} %{LOGLEVEL:loglevel} \s*\[%{DATA:logger}\]\s*%{GREEDYDATA:msg}\n*(?<stacktrace>(.|\r|\n)*)"}

i have checked it with http://grokconstructor.appspot.com/do/match. but got this matching error for stacktrace field.enter image description here

please do suggest some suggestions. thanks in advance.

Upvotes: 1

Views: 1548

Answers (1)

Phonolog
Phonolog

Reputation: 6511

You will need a multiline filter if you want to match the whole stacktrace. This multiline filter should work for you:

codec => multiline {
        pattern => "^%{TIME} "
        negate => true
        what => previous
    }

Explanation: Every line not starting with a timestamp (like 23:28:32.226) will be regocnized as part of the previous line. See also the docs on dealing with multilines.

Now to your pattern. Following works for me:

%{TIME:timestamp} %{LOGLEVEL:loglevel}  \[%{DATA:logger}\]   %{GREEDYDATA:message}\n(?<stacktrace>(.|\r|\n)*)

Pretty self explaining, I hope: Escaping braces like [ and ] with \[ and \], \n to match the newline. Also note the spaces between the entries.

For the last part (stacktrace) also see this question on how to match everything including newlines.

A full configuration could look something like this:

input {
  file {
    path => "/var/log/yourlog.log"
    start_position => "beginning"
    codec => multiline {
        pattern => "^%{TIME} "
        negate => true
        what => previous
    }
  }
}
filter {
  grok {
    match => [ "message", "%{TIME:timestamp} %{LOGLEVEL:loglevel}  \[%{DATA:logger}\]   %{GREEDYDATA:message}\n(?<stacktrace>(.|\r|\n)*)" ]
  }
}

Results on http://grokconstructor.appspot.com: results

Upvotes: 1

Related Questions