CODEWITHSUNDEEP
logstashlogstash-groklogstash-configuration
Nalini Wanjale
Nalini Wanjale

Reputation: 1757

Logstash filter for spring boot logs to capture loglevel,class name

I am trying to filter spring boot App logs using logstash using below configuration in filter block

filter {

 grok {
    match => [ "message",
               "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- \[(?<thread>[A-Za-z0-9-]+)\] [A-Za-z0-9.]*\.(?<class>[A-Za-z0-9#_]+)\s*:\s+(?<logmessage>.*)",
               "message",
               "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(?<logmessage>.*)"
             ]
  }


  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
  }
}

but it is not showing level,pid,class name in KIbana. enter image description here

Few Lines from log file

2020-01-23 12:08:51.468 ERROR 13216 --- [http-nio-8085-exec-1] com.poc.SampleLog.DemoController         : java.lang.NullPointerException

java.lang.NullPointerException: null
    at com.poc.SampleLog.DemoController.exception2(DemoController.java:36) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]

What should I change in grok filter to capture loglevel,classname etc. Thanks in advance.

Upvotes: 1

Views: 3814

Answers (3)

Anton
Anton

Reputation: 720

Such pattern works fine for default spring.log files

filter {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:time}\s+%{LOGLEVEL:log_level}\s+\[%{DATA:appName},%{DATA:traceId},%{DATA:spanId},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:messageTmp}" }
#      remove_field => "message"
    }

    date {
      match => ["time", "YYYY-MM-dd HH:mm:ss.SSS"]
      target => "@timestamp"
      remove_field => "time"
    }

    mutate {
      add_field => {"serviceName" => "back"}
      rename => {"messageTmp" => "message"}
    }
}

Upvotes: 0

Ishara Dayarathna
Ishara Dayarathna

Reputation: 3601

Try this:

INPUT:

2020-01-23 12:08:51.468 ERROR 13216 --- [http-nio-8085-exec-1] com.poc.SampleLog.DemoController         : java.lang.NullPointerException

java.lang.NullPointerException: null
    at com.poc.SampleLog.DemoController.exception2(DemoController.java:36) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]

GROK PATTERN:

(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- \[%{GREEDYDATA:thread}\] %{GREEDYDATA:class}: %{GREEDYDATA:exception}%{SPACE}(?<stacktrace>(.|\r|\n)*)

OUTPUT:

{
  "timestamp": [
    [
      "2020-01-23 12:08:51.468"
    ]
  ],
  "YEAR": [
    [
      "2020"
    ]
  ],
  "MONTHNUM": [
    [
      "01"
    ]
  ],
  "MONTHDAY": [
    [
      "23"
    ]
  ],
  "TIME": [
    [
      "12:08:51.468"
    ]
  ],
  "HOUR": [
    [
      "12"
    ]
  ],
  "MINUTE": [
    [
      "08"
    ]
  ],
  "SECOND": [
    [
      "51.468"
    ]
  ],
  "level": [
    [
      "ERROR"
    ]
  ],
  "pid": [
    [
      "13216"
    ]
  ],
  "BASE10NUM": [
    [
      "13216"
    ]
  ],
  "thread": [
    [
      "http-nio-8085-exec-1"
    ]
  ],
  "class": [
    [
      "com.poc.SampleLog.DemoController         "
    ]
  ],
  "exception": [
    [
      "java.lang.NullPointerException"
    ]
  ],
  "SPACE": [
    [
      "\n\n"
    ]
  ],
  "stacktrace": [
    [
      "java.lang.NullPointerException: null\n    at com.poc.SampleLog.DemoController.exception2(DemoController.java:36) ~[classes/:na]\n    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]\n    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]\n    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]\n    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]"
    ]
  ]
}

Upvotes: 3

apt-get_install_skill
apt-get_install_skill

Reputation: 2908

Since your messages span over multiple lines (like stacktraces generally do), you would need to add the multiline-flag m in your regex/grok-pattern (see this tutorial for example).

Parsing stacktraces is not an easy task. So I do not expect that by just adding the multiline-flag it will automatically work. You would need to debug it quite often I guess.

But in my opinion you definetly need to use the multiline-flag.

Upvotes: 0

Related Questions