CODEWITHSUNDEEP
elasticsearchlogstashlogstash-configuration
Heschoon
Heschoon

Reputation: 3019

Remove an event field and reference it in Logstash

Using Logstash, I want to index documents into Elasticsearch and specify the type, id etc of the document that needs to be indexed. How can I specify those in my config without keeping useless fields in my documents?

Example: I want to specify the id used for insertion:

input {
    stdin {
        codec => json {}
    }
}

output {
    elasticsearch { document_id => "%{[id]}" }
}

This will insert the document in Elasticsearch with the id id but the document will keep a redundant field "id" in the mapping. How can I avoid that?

I thought of adding 

filter{ mutate { remove_field => "%{[id]}"} }

in the config, but the field is removed and cannot consequently be used as document_id...

Upvotes: 2

Views: 1100

Answers (1)

Magnus Bäck
Magnus Bäck

Reputation: 11571

Right now this isn't possible. Logstash 1.5 introduces a @metadata field whose contents aren't included in what's eventually sent to the outputs, so you'd be able to create a [@metadata][id] field and refer to that in your output,

output {
    elasticsearch { document_id => "%{[@metadata][id]}" }
}

without that field polluting the message payload indexed to Elasticsearch. See the @metadata documentation.

Upvotes: 4

Related Questions