CODEWITHSUNDEEP
springelasticsearchspring-data-elasticsearch
James111
James111

Reputation: 15903

How to keep elasticsearch data in sync with MySql?

I'm using spring-data-jpa to persist data to MySql and I've recently implemented spring-data-elasticsearch to insert data into ES on INSERT (e.g new photo is uploaded).

Now obviously this only inserts the initial data, soon after the it will become stale.

Lets take instagram as an example:

  1. You post a photo
    • Photo (photo object - description, location, tags)
    • User (user who posted)
    • No likes (0)
    • No comments (0)

All this data would be inserted under a photo index in elasticsearch. When users start liking and commenting on the photo, the data will become stale.

How can we fix this?!

Would we create a cronjob/task which would go through all the photos and insert each of them into elasticsearch every X minutes (this would mean removing all current indexes for photo)?! This seems like it'd take a long time and use a lot of resources!

Instead of inserting all the data again, would we simply update all the current elasticsearch indexes?

What have you done to combat this issue?

Upvotes: 2

Views: 1330

Answers (1)

Alcanzar
Alcanzar

Reputation: 17155

The standard way to do this is to keep a last updated type of column in the database and update it whenever there is a change.

Your indexer then just needs to query out anything from the database since the last time you successfully indexed data.

Depending on how reliable you can make it, you may still need to do a full comparison periodically.

Upvotes: 3

Related Questions