CODEWITHSUNDEEP
elasticsearchlogstash
Kodo
Kodo

Reputation: 551

How to deduplicate documents while indexing into elasticsearch from logstash

I'm using Logstash 1.4.1 together with ES1.01 and would like to replace already indexed documents based on a calculated checksum. I'm currently using the "fingerprint" filter in Logstash which creates a "fingerprint" field based on a specified algorithm. Now - what I want to accomplish is that ES replaces an already existing document based on an identical fingerprint value.

Say, for example that I have a document with a fingerprint-field value of "2c9a6802e10fbcff36177e0b88993f90868fa6fa". Now - if a document with an identical fingerprint value is about to be indexed, I want it to replace the old document already present in the index.

I've tried to add the following to the "elasticsearch-template.json" template file which I assume is used by the Logstash ES-output plugin:

...
  "mappings" : {
    "_default_" : {
       "_id" : {"index": "not_analyzed", "store" : false, "path" : "fingerprint" },
       "_all" : {"enabled" : true},
       "dynamic_templates" : [ {
...

but it doesn't work. What am I doing wrong here?

Cheers

Upvotes: 9

Views: 8348

Answers (2)

John Petrone
John Petrone

Reputation: 27487

I would use the document_id parameter in your logstash elasticsearch output section:

document_id

Value type is string
Default value is nil

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id

I believe the entry should be something like this:

document_id => "%{fingerprint}"

It uses logstash's sprintf format to replace a string with the contents of a field:

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#sprintf

Upvotes: 15

ppearcy
ppearcy

Reputation: 2762

Assuming the fingerprint is getting set as the _id, you may be hitting an issue with logstash's daily index management and not using the timestamp from your data.

Ensure that you have your timestamp set from the input data, so you are guaranteed the document goes to the correct daily index:

http://logstash.net/docs/1.4.2/filters/date

If my guess is correct, you should see that your duplicate documents have different @timestamp and are in different daily indexes.

Upvotes: 3

Related Questions