CODEWITHSUNDEEP
parsingelasticsearchgroklogstash
user2359303
user2359303

Reputation: 271

Logstash date parsing error

Summary: I have my log with the date format as shown :

2013/05/09-05:19:16.772

Now I want to use logstash and send these logs to elastic search. But the problem is that I want that the timestamp value should be that of the logs and not the current time.

Therefore, I have written the following. This fails saying this:

Invalid format: "2013/05/09-05:19:16.876" is malformed at "/05/09-05:19:16.876", :backtrace=>["org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:866)"

My conf file is:

input {
  stdin {
      type => "stdin-type"
  }
}

filter {
  grok {
    type => "stdin-type"
    patterns_dir=>["./patterns"]
    pattern => "%{PARSE_ERROR}"
    add_tag=>"%{type1},%{type2},%{slave},ERR_SYSTEM"
  }

  date {
    type => "stdin-type"
    match=>["ts","yyyy/mm/dd-HH:mm:ss.SSS"]
    locale=>"en"
  }

  mutate {
    type=>"stdin-type"
    replace => ["@message", "%{message}" ]
    replace => ["@timestamp", "%{ts}" ]
  }
}

output {
  stdout { debug => true debug_format => "json"}
  elasticsearch { }
}

I am really stuck here. Need some expert help.

Thanks.

Upvotes: 2

Views: 5727

Answers (3)

Chakra Yadavalli
Chakra Yadavalli

Reputation: 399

You have specifed lowercase m instead of M. m is for minutes and M is for months.

Try:

yyyy/MM/dd-HH:mm:ss.SSS

Upvotes: 4

Richard Nienaber
Richard Nienaber

Reputation: 10564

What is not readily apparent from the error until you enable verbose mode (-vv) from the command line is that the 'date' filter is using the JodaTime library. In my case, I was using a regex from a previous step to parse the date and time. This obviously won't work because it's not what JodaTime expects. For the allowable date time formats, you should look at the DateTimeFormat documentation.

Upvotes: 1

Nikolay Bryskin
Nikolay Bryskin

Reputation: 21

Try

YYYY/MM/dd-HH:mm:ss.SSS

instead

Upvotes: 0

Related Questions