CODEWITHSUNDEEP
mysqljdbcelasticsearchelasticsearch-plugindatabus
Cassius
Cassius

Reputation: 153

Sync MySQL inserts, updates, and delete operations with ElasticSearch

Here is a rather common case for elasticsearch(ES): as ES is near realtime, we want to sync it with our MySQL as much close to real time as possible. The initial tables are imported into index and it is working fine, but now we need to track inserting, updating and deleting new records and here is the place we have a question to. Import is done by elasticsearch-jdbc importer. It can have an interval parameter to periodically poll data, but:

  1. we cannot add fields to db which will be responsible for fetching new/updated rows,
  2. we want to have more atomic operations like DB insert row->ES insert document, DB update row->ES update document and so on.

Two options which might meet our needs:

Other solutions are welcome!

Upvotes: 4

Views: 3079

Answers (1)

charisMao
charisMao

Reputation: 99

The Logstash with logstash-input-jdbc plugin just can sync insert and update operation. Bug delete operation can't be sync directly. We talked more about How to sync Mysql delete operation with logstash-input-jdbc? in https://discuss.elastic.co/t/delete-elasticsearch-document-with-logstash-jdbc-input/47490/9.

Upvotes: 1

Related Questions