Logstash '_grokparsefailure' issue

Question

I am using custom grok pattern to parse my log file, Not matter what i do it always give me the _grokparsefailure exception.

However, it work perfectly on https://grokdebug.herokuapp.com/

My custom pattern file is located at,

C:\Users\Username\projects\Logstash\bin\patterns

Filename: mylogpattern

    LogLevel [I|V|D|M|W|E|A|F]
    MODULE \b\w+\b|------
    MESSAGEID (?:[+-]?(?:[0-9]+))|----
    SUBMODULE (.*?:)
    MESSAGE (.*)|(.*?:)|(.*\s*?:)

My Logstash Config File Looks like this:

input{
    beats{
      host => "192.168.56.1"
      port => 7088
      congestion_threshold => 200
    }
}
filter {
    if [type] == "MyLog"{
        grok{
          patterns_dir => ["C:\Users\Username\projects\Logstash\logstash\bin\patterns"]
          match => { "message" => "%{YEAR:Year}%{MONTHNUM:Month}%{MONTHDAY:Day} %{HOUR:Hour}%{MINUTE:Minute}%{SECOND:Second} %{LogLevel:LogVerbosity} %{MODULE:MODULENAME}%{SPACE}%{MESSAGEID:MESSAGEID} %{SUBMODULE:SUBMODULE} %{MESSAGE:MESSAGE}" }
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
        }
    }  
}
output {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
}

Sample Log File:

20160509 115108 I GEN 0000 ConnectionHandler.reconnect: Reconnect to the DB was done.

20160509 115108 I GEN 0000 84.1.3.1000012 : Reconnect to the DB was done.

It works perfectly on grok debugger, but somehow it is failing while parsing.

tags:beats_input_codec_plain_applied, _grokparsefailure 

Could someone please help me out with this, What am I doing wrong?

EDIT 1 :

I can see, for first few seconds, Logs are getting parsed. But , After that it is throwing the error.

All log follows essentially the same format, also i have added GREEDYDATA filter for the long messages, it is able to parse it successfully. I can still see some parsed logs in kibana.

Its just the log count is really high and log file also have continuous blank lines in between.

Does it try to parse those blank lines as well ?

Answer 1

i think there is a mistake at your grok paterns_dir

try this one : patterns_dir => "./patterns"