CODEWITHSUNDEEP
elasticsearchlogstashlogstash-configurationlogstash-file
robbmj
robbmj

Reputation: 16526

Writing to @timestamp in LogStash

I need to write the value of a UNIX timestamp field to @timestamp so that I can correctly index data flowing through logstash, I have this part working. However I also have the requirement that @timestamp's value should be the insertion time. To this end I have made a temporary field that holds @timestamps original value.

Here is what I am working with:

filter {
    csv {
        separator => "  " # <- this white space is actually a tab, don't change it, it's already perfect
        skip_empty_columns => true
        columns => ["timestamp", ...]
    }
    # works just fine
    mutate {
        add_field => {
            "tmp" => "%{@timestamp}"
        }
    }
    # works just fine
    date {
       match => ["timestamp", "UNIX"]
       target => "@timestamp"
    }
    # this works too
    mutate {
        add_field => {
            "[@metadata][indexDate]" => "%{+YYYY-MM-dd}"
        }
    }   
    # @timestamp is not being set back to its original value
    date {
        match => ["tmp", "UNIX"]
        target => "@timestamp"
    }
    # works just fine
    mutate {
        remove_field => ["tmp"]
    }
}

output {
    elasticsearch {
        hosts => "elasticsearch:9200"
        # this works
        index => "indexname-%{[@metadata][indexDate]}"
    }
}

The Problem is here:

date {
    match => ["tmp", "UNIX"]
    target => "@timestamp"
}

@timestamp is not being set back to its original value. When I check the data it has the same value as the timestamp field.

Upvotes: 0

Views: 1727

Answers (1)

Alcanzar
Alcanzar

Reputation: 17165

When you add the date to tmp, it gets added in ISO8601 format, so you need to use:

date {
    match => ["tmp", "ISO8601"]
    target => "@timestamp"
}

Upvotes: 2

Related Questions