CODEWITHSUNDEEP
elasticsearchlogstashdeduplication
cfiske
cfiske

Reputation: 194

How do I keep a count of deduplicated messages from Logstash in ElasticSearch?

I see from this question that document_id can easily be used in Logstash to replace a duplicate record in ElasticSearch. How would I add/increment a count value for e.g. repeating syslog messages? Instead of just replacing the record I want to increment the count so I know how many times it has repeated.

Upvotes: 3

Views: 792

Answers (2)

cfiske
cfiske

Reputation: 194

I think I have found what I need. An upsert will insert if a record doesn't exist, and update if it does. And enabling the _source field will allow incrementing of an existing field. That combination gives me the ability to start with a count of 1, and increment by 1 if the record already exists. Thanks @Alcanzar, your answer got my brain going in the right direction to find this.

Upvotes: 1

Alcanzar
Alcanzar

Reputation: 17155

Depending on what you are using to view the data, it might just be as simple as looking at the _version field of the documents. ES will automatically update that value when something changes for the document. Kibana doesn't show the _version field (https://github.com/elasticsearch/kibana/issues/1234), but it's there.

Upvotes: 2

Related Questions