CODEWITHSUNDEEP
logstash
user3024742
user3024742

Reputation: 63

Logstash: Failed parsing date

Summary:
logstash -> elasticsearch --> Failed parsing date shown in debug output
Events in logfile contain field @timestamp (format: 2014-06-18T11:52:45.370636+02:00)
Events are actually processed to elasticsearch but 'failed parsing' errors are shown.

Versions:
Logstash 1.4.1
Elasticsearch 1.20

Is there something I do wrong?

I have log files that contain events like this:

{"@timestamp":"2014-06-18T11:52:45.370636+02:00","Level":"Info","Machine":"X100","Session":{"MainId":0,"SubId":"5otec"},"Request":{"Url":"http://www.localhost:5000/Default.aspx","Method":"GET","Referrer":"http://www.localhost:5000/Default.aspx"},"EndRequest":{"Duration":{"Main":0,"Page":6720}}}

I use this logstash config:

input {
  file {
    path => "D:/testlogs/*.log"
    codec => "json"
    start_position => [ "beginning" ]
  }
}
filter {
  date { 
    match => [ "@timestamp", "ISO8601" ]
    }
}
output {
  stdout { 
    codec => json 
    }
  elasticsearch {
    protocol => http
    host => "10.125.26.128"
  }
}

When I run logstash with this config on the events in the log files I get the following error:

[33mFailed parsing date from field {:field=>"@timestamp", :value=>"2014-06-18T12:18:34.717+02:00", :exception=>#<TypeError: cannot convert instance of class org.jruby.RubyTime to class java.lang.String>

Now the thing is that actually the events are imported in elasticsearch, but I see these errors.
Can this be a problem or can these failed parsing errors being ignored?

Upvotes: 2

Views: 2284

Answers (1)

Ban-Chuan Lim
Ban-Chuan Lim

Reputation: 7890

You logs are already in json format, so you no need to parse the date. You can use the json filter to parse all the field and value.

For example, with this configuration:

input {
    file {
       path => "D:/testlogs/*.log"
       codec => "json"
    }
}

filter {
    json {
        source => "message"
    }
}

output {
    stdout {
            codec => rubydebug
    }
}

I can parse your log successfully. And the output is :

{
"@timestamp" => "2014-06-18T17:52:45.370+08:00",
     "Level" => "Info",
   "Machine" => "X100",
   "Session" => {
    "MainId" => 0,
     "SubId" => "5otec"
},
   "Request" => {
         "Url" => "http://www.localhost:5000/Default.aspx",
      "Method" => "GET",
    "Referrer" => "http://www.localhost:5000/Default.aspx"
},
"EndRequest" => {
    "Duration" => {
        "Main" => 0,
        "Page" => 6720
    }
},
  "@version" => "1",
      "host" => "ABC",
      "path" => "D:/testlogs/*.log"
}

I also have try that the codec => json in file input is not work but is work in stdin input. Maybe this is a bug in logstash.

Hope this can help you.

Upvotes: 1

Related Questions