CODEWITHSUNDEEP
elasticsearchlogstash
JeyJ
JeyJ

Reputation: 4080

Some of KV filter values has custom date that identified as string in Kibana

I'm using kv filter in Logstash to process config file in the following format :

key1=val1
key2=val2
key3=2020-12-22-2150

with the following lines in Logstash :

kv {
      field_split => "\r\n"
      value_split => "="
      source => "message"
    }

Some of my fields in the conf file have a the following date format : YYYY-MM-DD-HHMMSS. When Logstash send the fields to ES, Kibana display them as strings. How can I let Logstash know that those fields are date fields and by that indexing them in ES as dates and not strings ?

I don't want to edit the mapping of the index because it will require reindexing. My final goal with those fields is to calculate the diff between the fields (in seconds, minutes,hours..) and display it in Kibana.

The idea that I have :

  1. Iterate over k,v filter results, if the value is of format YYYY-MM-DD-HHMMSS (check with regex)
  2. In this case, chance the value of the field to milliseconds since epoch

I decided to use k,v filter and Ruby code as a solution but I'm facing an issue.

Upvotes: 0

Views: 239

Answers (2)

Jaycreation
Jaycreation

Reputation: 2089

It could be done more easily outside of logstash by adding a dynamic_template on your index and let him manage field types. You can use the field name as a detector if it is clear enough (*_date) or define a regex

"match_pattern": "regex",
"match": "^(0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])[- /.](19|20)\d\d$"

The code above hasnot been tested.

You can find the official doc here. https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html

Upvotes: 1

JeyJ
JeyJ

Reputation: 4080

My solution :

  1. I used the kv filter to convert each line into key value set.
  2. I saved the kv filter resut into a dedicated field.
  3. On this dedicated field, I run a Ruby script that changed all the dates with the custom format to miliseconds since epoch.

code :

filter {
  if "kv_file" in [tags]  {
    kv {
      field_split => "\r\n"
      value_split => "="
      source => "message"
      target => "config_file"
    }

    ruby {
      id => "kv_ruby"
      code => "
        require 'date'
        re = /([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])-[0-23]{2}[0-5]{1}[0-9]{1}[0-5]{1}[0-9]{1})/
        hash = event.get('config_file').to_hash
        hash.each { |key,value|
          if value =~ re
            date_epochs_milliseconds = DateTime.strptime(value,'%F-%H%M%S').strftime('%Q')
            event.set(key, date_epochs_milliseconds.to_i)
          end
        }
      "
    }
  }
}

By the way, if you are facing the following error in your Ruby compilation : (ruby filter code):6: syntax error, unexpected null hash it doesn't actually mean that you got a null value, it seems that it is related to the escape character of the double quotes. Just try to replace double quotes with one quote.

Upvotes: 0

Related Questions