CODEWITHSUNDEEP
javalogstashkibana-4elastic-stack
dbaq
dbaq

Reputation: 1357

ELK: How to group stacktraces by exception class in Kibana

I am setting up the ELK stack for a java web application. I successfully parsed multiline java stacktraces with logstash and display the count of exceptions in kibana. Now I would like to display a date histogram with the count of exceptions group by exception class, i.e. 2 java.lang.NullPointerException, 3 java.lang.ArithmeticException per minutes or seconds.

In kibana, I can see the full stacktrace indexed. But I wasn't able to visualize my exceptions group by classes. What is the best practice here? Try to retrieve the fully qualified class name with Logstash and do a term filter in kibana? or is there a way to use the power of ES in kibana?

An example of the beginning of a message field:

2015-08-15 23:23:51.695 [qtp1010279661-1074] ERROR c.m.w.s.proxies.ProxyServlet:71 - Can't get content from url http://localhost:8080/...
org.apache.http.conn.HttpHostConnectException: Connect to localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1, localhost/fe80:0:0:0:0:0:0:1%1] failed: Connection refused
    at org.apache.http.impl.conn.HttpClientConnectionOperator.connect...

My logstash configuration:

input {
    file {
        path => "/Users/dbaq/web-app.log"
        start_position => beginning
    }
}

filter {
    multiline {
        pattern => "%{TIMESTAMP_ISO8601:timestamp}"
        negate => true
        what => "previous"
    }

    grok {
        match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:thread}\]\s*%{LOGLEVEL:severity}\s*%{DATA:class}:%{NUMBER:line:int}\s*\- %{GREEDYDATA:message}"]
        overwrite => [ "message" ]
    }

    date {
        match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
    }
}

output {
    elasticsearch {
        protocol => "http"
    }
    stdout {}
}

Thanks for your help

EDIT 1: My class field in my logstash pattern represents the class where the exception was thrown, in my example: c.m.w.s.proxies.ProxyServlet. I want to aggregate by my Exception class: org.apache.http.conn.HttpHostConnectException.

Upvotes: 3

Views: 7473

Answers (2)

Marvin
Marvin

Reputation: 81

As @Alain Collins already pointed out you can use a "data table" for visualization.

I would suggest that you are going to add a multiline codec to your input with the following pattern:

input {
    file {
         path => "/Users/dbaq/web-app.log"
         start_position => beginning
         codec => multiline {
              pattern => "^\s"
              what => "previous"
         }
    }
}

What you can then do is using the predefined grok regex JAVASTACKTRACEPART by doing this:

if "multiline" in [tags] {
    grok {
        match => ["message", "%{JAVASTACKTRACEPART}"]
    }
}

Please note that this will also create a field called class. You can use this field to perform a terms search and get your count metric applied to it.

Upvotes: 4

Alain Collins
Alain Collins

Reputation: 16362

A "data table" visualization type should allow you to aggregate ("group by") your class field and show the count.

EDIT: oops, wrong field.

In your original grok, you put everything after the line number back into message. To extract the exception location from this string, you'll want another grok stanza.

What the pattern looks like depends on how consistent the error messages are. In your example, it looks like "hyphen description ... location colon". If they're all like that, you can make a pattern to match.

Upvotes: 0

Related Questions