CODEWITHSUNDEEP
elasticsearch
Lama
Lama

Reputation: 2966

Avoid duplicate documents in Elasticsearch

I parse documents from a JSON, which will be added as children of a parent document. I just post the items to the index, without taking care about the id.

Sometimes there will be updates to the JSON and items will be added to it. So e.g. I parsed 2 documents from the JSON and after a week or two I parse the same JSON again. This time the JSON contains 3 documents.

I found answers like: 'remove all children and insert all items again.', but I doubt this is the solution I'm looking for.

I could compare each item to the children of my target-parent and add new documents, if there is no equal child.

I wondered if there is a way, to let elasticsearch handle duplicates.

Upvotes: 3

Views: 13560

Answers (2)

Alex
Alex

Reputation: 43

When adding a new document to elasticsearch, it first scans the existing documents to see if any of the IDs match. If there is already an existing document with that ID, the document will be updated instead of adding in a duplicate document (the version field will be updated at the same time to track the amount of updates that have occurred). You will therefore need to keep track of your document IDs somehow and maintain the same IDs throughout matching documents to eliminate the possibility of duplicates.

Upvotes: 0

Vineeth Mohan
Vineeth Mohan

Reputation: 19253

Duplication needs to be handled in ID handling itself. Choose a key that is unique for a document and make that as the _id. In the the key is too large or it is multiple keys , create a SHAH checksum out of it and make that as the _id.

If you already have dedupes in the database , you can use terms aggregation nested with top_hits aggregation to detect those.

You can read more about this approach here.

Upvotes: 7

Related Questions