CODEWITHSUNDEEP
elasticsearchlogstashkibanagrok
user1137582
user1137582

Reputation: 699

Grok filter not working even though it works in grok debugger

I'm using logstash-1.4.0 with elasticsearch 1.3.4 and kibana 3.1.1 (I know I'm outdated, that's the best I can do right now).

Log Example:

2016-05-31 16:05:33 RequestManager [INFO] The manual flag LOLROFLin TRALALA 123456Was changed to true

My grok filter:

filter {
    grok {
        match => { "message" => "%{DATESTAMP:timestamp} %{WORD:clazz} %{NOTSPACE:level} %{GREEDYDATA:content}"}
    }

    if (!([stack_trace])) and (!([clazz] == "RequestAsset")) {
        drop {}
    }
}

My questions are:

  1. Why do I not see the grok fields in kibana? I only see the default fields but not mine. Grok Debugger shows success, but kibana does not work.

  2. My goal is to drop any log message that does not have a stack trace OR is not from class (called clazz in my grok filter) "RequestAsset". Should this work? can I use the fields created by the grok filter in a seperate if filter?

EDIT: I realised what went wrong, I was using the log4j plugin which already seperates the log to its contents, and the field message was already just the message itself.

Upvotes: 0

Views: 1735

Answers (2)

user1137582
user1137582

Reputation: 699

I realised what went wrong, I was using the log4j plugin which already seperates the log to its contents, and the field message was already just the message itself.

Upvotes: 0

Jeroen Vandevelde
Jeroen Vandevelde

Reputation: 456

I tested your grok filter in this grok debugger and it failed. So i have rewritten it.

Here is the correct grok filter.

filter {
grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:clazz} %{NOTSPACE:level} %{GREEDYDATA:content}"}
}

if (!([stack_trace])) and (!([clazz] == "RequestAsset")) {
    drop {}
}

TIMESTAMP_ISO8601 => %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

If you see "_grokparsefailure" in Kibana, you know that your grok filter failed.

On your second question shouldn't you use the OR operator?

Upvotes: 0

Related Questions