CODEWITHSUNDEEP
databaseelasticsearchapache-kafkaapache-kafka-connect
user404
user404

Reputation: 2028

Kafka to ES and other sink DB

Using a mysql source connector, I can capture the mysql change and post to them ES or another db for backup. But for that I need separate connector(both source and sink) for each table I have in my source db. So my question is:

Without creating same amount of source and sink connector for each tables, how can I achieve the same purpose? As this is cumbersome to create that many connectors. So that backing up db(replica) and making faster response service for clients would become much easier for me. Or there is no way to do this?

Upvotes: 1

Views: 49

Answers (1)

Giorgos Myrianthous
Giorgos Myrianthous

Reputation: 39840

For the source connectors, you can use table.whitelist. For example, 

table.whitelist: "User, Address, Email"

Sink connectors can only be configured for one table at a time.

And I wouldn't say that it is hard to maintain multiple sink/source connectors and topics. From my experience, it is harder to maintain connectors which are replicating data from multiple topics/sources. For example, if you want to apply SMT (Simple Message Transform) on a particular topic, you won't be able to achieve it if you don't have isolated connectors as SMTs are applied on a connector level. Furthermore, if you configure a single connector for all of your sources and at some point it fails, all of your target systems will encounter downtime.

Upvotes: 1

Related Questions