CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectdebezium
JDev
JDev

Reputation: 1832

Kafka Connect: Single connector or connector per table approach

I have a database say test and we are having multiple Kafka Debezium Connectors on it. Each connector is associate with one table.

My question is in terms of memory usage, which is a better approach:

  1. One connector per database OR
  2. One connector per table

Upvotes: 1

Views: 1145

Answers (2)

Tudor Plugaru
Tudor Plugaru

Reputation: 397

I think it really depends on your use case. I don't think there is a general approach for all the usecases. For example, at my current job, we decided to have 4 connectors that stream changes from the same database, but, each of them is streaming from a subset of tables. The main reason is that we don't want to have a single point of failure where a bad record inside DB can break all our usecases that rely on CDC, hence, we divided the tables and assigned them to a connector. Note that it's not good to have a lot of replication slots on the database also. So it really depends on your usecase.

Upvotes: 1

ChristDist
ChristDist

Reputation: 768

Considering all performance factors, it is always a recommended approach to have a single source connector (multiple instances to share the load), with replicator or configuration file per database instance (test1, test2, test3 etc), having multiple tables, so the data ingress would be 1 table -> 1 topic.

You can have a better view at Oracle Golden Gate implementation pattern for the same.

https://rmoff.net/2018/12/12/streaming-data-from-oracle-into-kafka/

Upvotes: 2

Related Questions