CODEWITHSUNDEEP
logstashlogstash-grok
Antoine
Antoine

Reputation: 5245

Logstash grok multiple matches failure

I have these logs where I'm trying to extract multiple information form each line with grok. But it seems some matches are not taken into account, although they seem fine to me.

Here's a sample line:

"#wtb# GetSP ok. Referer:http://test.redacted.com/path?query=string. Wtb:535e2554bdfdf33a22f564d0. Name:Client. Eans:3017565410073."

And heres' the related part of the conf file:

grok{
  break_on_match => false
  match => [
    "msg", "Referer:%{URI:referer}\.",
    "msg", "Wtb:%{WORD:wtb}",
    "msg", "Name:(?<name>[^\.]+)",
    "msg", "Eans:(?<eans>[\d,]+)",
    "referer", "https?://%{HOSTNAME:host}"
  ]
  tag_on_failure => []
}

I'm using the multiple matches because each line can be any combination of the various parameters given in sample.
In Kibana, the events have the referer and host fields added, but all others are missing (wtb, name, eans). I have no idea why. It does not stop after a successful match since the last pattern is added. Can anyone spot what I'm missing?

Upvotes: 2

Views: 2866

Answers (1)

Alcanzar
Alcanzar

Reputation: 17155

The simple answer is that grok doesn't work like that. The way it's implemented, it only matches the first pattern in your grok list for a given key.

If you look at the source code of jls-grok-0.10.12/lib/grok/pure/pile.rb, you'll see this:

def match(string)
  @groks.each do |grok|
    match = grok.match(string)
    if match
      return [grok, match]
    end
  end
  return false
end # def match

which basically causes it to stop the grok pile for a given key after the first match.

So to do what you want, you are going to need to break your grok so that there's only one msg pattern per grok.

Upvotes: 5

Related Questions