CODEWITHSUNDEEP
timestamphistogramkibana
halpsb
halpsb

Reputation: 1106

Kibana: Use timestamp from log file instead of @timestamp

I am trying to use the timestamps that are in my log files. However, these timestamps have the following format, that is different from logstash's @timestamp field.

2014-10-31 02:45:09,355

When attempting to use my field related to the timestamps in my log files, the histogram is not displayed. I have changed the "Time field" in the "Timepicker" tab of my dashboard's settings accordingly, as well as in the histogram's settings.

Any suggestion will be greatly appreciated.

Edit:

Below is what I currently have in my filter: 

filter {
    grok {
        patterns_dir => ".\patterns"
        match => [ "message", "%{LOGTIMESTAMP:tstamp}" ]
    }
    date {
        locale => "en"
        timezone => "Europe/Paris"
        match => [ "tstamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
        target => "tstamp"
    }
}

As for my regular expression: 

LOGTIMESTAMP (\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})

With this configuration, I obtain the following error: "Failed parsing date from field"

Any ideas?

Upvotes: 2

Views: 9183

Answers (4)

snket_d
snket_d

Reputation: 1

I had a similar problem for the same log pattern "2014-10-31 02:45:09,355". This configuration worked for me.

grok {
    match => [ "message", "%{DATESTAMP:timestamp}" ]
}
date {
    locale => "en"
    timezone => "Europe/Paris"
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
    target => ["tstamp"]
    remove_field => ["timestamp"]
}

Upvotes: 0

s1d
s1d

Reputation: 540

Took me some time to figure it out but the following format works for me for searching entries between specific times (2015 May 27 11am to 1130am to be exact)-

@timestamp:[1432704600000 TO 1432706400000]

Kibana requires timestamp in milliseconds from epoch format.

Upvotes: 0

rabbbit
rabbbit

Reputation: 11

You have to use "date" filter to tell Logstash about the date format - it will then convert your timestamp from the file into the @timestamp field.

Take a look here: http://logstash.net/docs/1.4.2/filters/date

Upvotes: 0

williamthomas
williamthomas

Reputation: 105

Id like to have added this as comment, but I don't have the reputation... anyway, I'm guessing that Kibana thinks your datetime is a string, not a type date (check the mapping,
http://localhost:9200/_mapping?pretty or whatever).

I had a similar problem, but followed this: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-timestamp-field.html

However, any mapping I did didn't work, until I output it in this exact format 2014-12-03T09:04:02. (If I dropped the T it wouldn't work)

My advice is try to transform that log file time to match as I wasnt able to get Kibana to take any other format.

Upvotes: 0

Related Questions