CODEWITHSUNDEEP
elasticsearchlogstashelastic-stackkibana-4
RS_ingo
RS_ingo

Reputation: 429

Not able to parse string to date in logstash/elasticSearch

I had created a logstash script to read a logfile which is having various timestamp of format "2018-05-08T12:18:53.506+0530". I am trying to parse it to date using the date filter in log stash

date{
     match => ["edrTimestamp","yyyy-MM-dd'T'HH:mm:ss.SSS'Z'","ISO8601"]
     target => "edrTimestamp"
}

The running the above logstash script it creates a elastic search index. But still the string is not parsed to date. It is also showing date parsed exception in the index.

It creates output like this.

{
          "tags": [
            "_dateparsefailure"
          ],
          "statusCode": "805",
          "campaignRedemptionLimitTotal": 1000,
          "edrTimestamp": "2018-05-22T16:41:25.162+0530 ",
          "msisdn": "+919066231327",
          "timestamp": "2018-05-22T16:41:25.122+0530",
          "redempKeyword": "print1",
          "campaignId": "C910101-1527004962-1582",
          "category": "RedeemRequestReceived"
        }

Please tell me whats wrong in the above code> I had tried many others alternative but still it is not working.

Upvotes: 1

Views: 805

Answers (2)

TheFiddlerWins
TheFiddlerWins

Reputation: 922

I don't think you should be escaping the Z. So you probably want something like:

yyyy-MM-dd'T'HH:mm:ss,SSS

Also you should not be using "Z" since your time is not Zulu (0 offset). You will want to contain the offset as part of the pattern. The Heroku grok debug app is useful for this.

If I pass your string

2018-05-08T12:18:53.506+0530

and use the filter %{TIMESTAMP_ISO8601} then it matches, this pattern is made up of the following sub-patterns:

 TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

Upvotes: 1

Alcanzar
Alcanzar

Reputation: 17165

Your issue is that your timestamp has a space at the end of it "edrTimestamp": "2018-05-22T16:41:25.162+0530 ", which is causing the date parsing to fail. You need to add a:

mutate {
  strip => "edrTimestamp"
}

before your date filter.

Upvotes: 2

Related Questions