CODEWITHSUNDEEP
logstashlogstash-grok
Dennis
Dennis

Reputation: 2964

What is the correct way to have multiple grok patterns in a Logstash filter?

What is the valid logstash config of these 2 options? 

else if [pipeline] == "tomcat_all" {
  grok {
    match => [ "message", "%{MONTH}%{SPACE}%{MONTHDAY},%{SPACE}%{YEAR}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}(?:AM|PM)%{SPACE}%{NOTSPACE:class}%{SPACE}%{NOTSPACE:type_log}%{SPACE}%{WORD:loglevel}:%{SPACE}%{GREEDYDATA:log_text}" ]
    match => [ "message", "%{TIME:timestamp}%{SPACE}\|-%{WORD:loglevel}%{SPACE}in%{SPACE}%{NOTSPACE:class}%{SPACE}%{GREEDYDATA:log_text}" ]

...

else if [pipeline] == "123" {
  grok {
    match => [ "message", "%{MONTH}%{SPACE}%{MONTHDAY},%{SPACE}%{YEAR}%{SPACE}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}(?:AM|PM)%{SPACE}%{NOTSPACE:class}%{SPACE}%{NOTSPACE:type_log}%{SPACE}%{WORD:loglevel}:%{SPACE}%{GREEDYDATA:log_text}" ]
  }
  grok {
    match => [ "message", "%{TIME:timestamp}%{SPACE}\|-%{WORD:loglevel}%{SPACE}in%{SPACE}%{NOTSPACE:class}%{SPACE}%{GREEDYDATA:log_text}" ]
  }

Logstash seems to start fine with both configurations and report no errors, but the grok parsing isn't working properly with multiple grok patterns yet.

Upvotes: 4

Views: 14291

Answers (1)

Sufiyan Ghori
Sufiyan Ghori

Reputation: 18753

In comparison, both will almost perform equally since the default value for break_on_match is true.

break_on_match

  • Value type is boolean
  • Default value is true

Break on first match. The first successful match by grok will result in > the filter being finished. If you want grok to try all patterns (maybe you are parsing different things), then set this to false.

Your first pattern can be further simplified as follows,

filter {
   grok {
     match => [ "message", "PATTERN1", "PATTERN2" ]
    }
}

Please refer to this answer as well, Multiple patterns in one log

Upvotes: 6

Related Questions