CODEWITHSUNDEEP
rubyindexingelasticsearchtimezonelogstash
Davmrtl
Davmrtl

Reputation: 178

Convert timestamp timezone in Logstash for output index name

In my scenario, the "timestamp" of the syslog lines Logstash receives is in UTC and we use the event "timestamp" in the Elasticsearch output:

output {
    elasticsearch {
        embedded => false
        host => localhost
        port => 9200
        protocol => http
        cluster => 'elasticsearch'
        index => "syslog-%{+YYYY.MM.dd}"
    }
}

My problem is that at UTC midnight, Logstash sends log to different index before the end of the day in out timezone (GMT-4 => America/Montreal) and the index has no logs after 20h (8h PM) because of the "timestamp" being UTC.

We've done a work arround to convert the timezone but we experience a significant performance degradation:

filter {
    mutate {
        add_field => {
            # Create a new field with string value of the UTC event date
            "timestamp_zoned" => "%{@timestamp}"
        }
    }

    date {
        # Parse UTC string value and convert it to my timezone into a new field
        match => [ "timestamp_zoned", "yyyy-MM-dd HH:mm:ss Z" ]
        timezone => "America/Montreal"
        locale => "en"
        remove_field => [ "timestamp_zoned" ]
        target => "timestamp_zoned_obj"
    }

    ruby {
        # Output the zoned date to a new field
        code => "event['index_day'] = event['timestamp_zoned_obj'].strftime('%Y.%m.%d')"
        remove_field => [ "timestamp_zoned_obj" ]
    }
}

output {
    elasticsearch {
        embedded => false
        host => localhost
        port => 9200
        protocol => http
        cluster => 'elasticsearch'
        # Use of the string value
        index => "syslog-%{index_day}"
    }
}

Is there a way to optimize this config?

Upvotes: 6

Views: 17455

Answers (5)

sastorsl
sastorsl

Reputation: 2135

Similar use case - but using the logstash file output plugin and writing files dated by the local time of the arrival of the event. Verified on logstash version 7.12.

Adapted from discuss.elastic.co, mainly zero padding the offset hours. NB! If your offset has half hours you will need to adjust accordingly.

filter {
    ruby {
        code => "
        require 'tzinfo'
        tz = 'Europe/Oslo'
        offset = TZInfo::Timezone.get(tz).current_period.utc_total_offset / (60*60)
        event.set('[@metadata][local_date]',
                  event.get('@timestamp').time.localtime(
                      sprintf('+%02i:00', offset.to_s)
                  ).strftime('%Y%m%d'))
        "
    }   
    if ([agent][type] == "filebeat") {
        mutate {
            add_field => ["file_path", "%{[host][name]}_%{[log][file][path]}.%{[@metadata][local_date]}"]
        }  
    } else {   
        mutate {
            add_field => ["file_path", "%{[agent][hostname]}_%{[agent][type]}.%{[@metadata][local_date]}"]
        }
    }
}

Upvotes: 0

user8223430
user8223430

Reputation: 31

In logstash version 5.0 and later, you can use this:

filter{
ruby {
        code => "event.set('index_day', event.get('[@timestamp]').time.localtime.strftime('%Y%m%d'))"
    }
}

Upvotes: 3

Lancger
Lancger

Reputation: 1

In Logstash Version 5.0.2，The API was modified. We can convert timestamp by local timezone for the index name. Here is my configuration:

filter { 
   ruby { 
       code => "event['index_day'] = event.timestamp.time.localtime.strftime('%Y.%m.%d')" 
   } 
}

Upvotes: 0

Jerry Zhu
Jerry Zhu

Reputation: 46

In version 1.5.0, we can convert timestamp by local timezone for the index name. Here is my configuration:

filter {
    ruby {
        code => "event['index_day'] = event.timestamp.time.localtime.strftime('%Y.%m.%d')"
    }
}
output {
    elasticsearch {
        host => localhost
        index => "thrall-%{index_day}"
    }
}

Upvotes: 1

Ban-Chuan Lim
Ban-Chuan Lim

Reputation: 7890

This is the optimize config, please have a try and test for the performance.

You no need to use mutate and date plugin. Use ruby plugin directly. 

input {
    stdin {
    }
}

filter {
    ruby {
            code => "
                    event['index_day'] = event['@timestamp'].localtime.strftime('%Y.%m.%d')
            "
    }
}

output {
    stdout { codec => rubydebug }
}

Example output:

{
       "message" => "test",
      "@version" => "1",
    "@timestamp" => "2015-03-30T05:27:06.310Z",
          "host" => "BEN_LIM",
     "index_day" => "2015.03.29"
}

Upvotes: 11

Related Questions