CODEWITHSUNDEEP
elasticsearchlogstashkibana
Jeremie Myara
Jeremie Myara

Reputation: 33

Grok configuration ELK

I have an original type of log to parse. The syntax is :

2013-01-05 03:29:38,842 INFO  [ajp-bio-8009-exec-69] web.CustomAuthenticationSuccessHandler - doLogin : admin.ebusiness date : 2013-01-05 03:29:38

When I use the grok pattern : 

if [type] in ["edai"] {
            grok {
            match => { "message" => ["%{YEAR:year}-%{WORD:month}-%{DATA:day} %{DATA:hour}:%{DATA:minute}:%{DATA:second},%{DATA:millis} %{NOTSPACE:loglevel} {0,1}%{GREEDYDATA:message}"] }
            overwrite => [ "message" ]
        }
    }

The pattern work as you can see, but when I go into Kibana, the log stay in one block in the "message" section like this:

2013-01-05 23:27:47,030 INFO [ajp-bio-8009-exec-63] web.CustomAuthenticationSuccessHandler - doLogin : admin.ebusiness date : 2013-01-05 23:27:47

I would prefer to have it like this:

{ "year": [["2013"]], "month": [["01"]], "day": [["05"]], "hour": [["04"]], "minute": [["04"]], "second": [["39"]], "millis": [["398"] ], "loglevel": [ ["INFO"]] }

Can you help me to parse it correctly please?

Upvotes: 1

Views: 107

Answers (1)

Evaldas Buinauskas
Evaldas Buinauskas

Reputation: 14077

Just tested this configuration. I kinda copied everything from your question.

input {
  stdin { type => "edai" }
}

filter {
  if [type] == "edai" {
    grok {
      match => { "message" => ["%{YEAR:year}-%{WORD:month}-%{DATA:day} %{DATA:hour}:%{DATA:minute}:%{DATA:second},%{DATA:millis} %{NOTSPACE:loglevel} {0,1}%{GREEDYDATA:message}"] }
      overwrite => [ "message" ]
    }
  }
}

output {
  stdout { codec => rubydebug }
}

This is the output:

{
          "year" => "2013",
       "message" => " [ajp-bio-8009-exec-69] web.CustomAuthenticationSuccessHandler - doLogin : admin.ebusiness date : 2013-01-05 03:29:38\r",
          "type" => "edai",
        "minute" => "29",
        "second" => "38",
    "@timestamp" => 2017-06-29T08:19:08.605Z,
         "month" => "01",
          "hour" => "03",
      "loglevel" => "INFO",
      "@version" => "1",
          "host" => "host_name",
        "millis" => "842",
           "day" => "05"
}

Everything seems fine from my perspective.

I had issue when I compared type the way you did:

if [type] in ["eday"]

It did not work and I've replaced it with direct comparison:

if [type] == "edai"

Also this worked too:

if [type] in "edai"

And that solved the issue.

Upvotes: 2

Related Questions