CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
naikum
naikum

Reputation: 71

Kafka Connect Design suggestion

Please suggest or guide me to the design of Apache Kafka connect for the scenario:

Source MySQL: 100GB , approx 300+ tables Target: PostgreSQL

Please let me know how many source connectors, topics, partion per topic and number of sink connectors need to design approximately.

Do I need to use Debezium connectors only?

Upvotes: 0

Views: 74

Answers (1)

ChristDist
ChristDist

Reputation: 768

As Apache Community recommended, The https://debezium.io/ Project provides a collection of high-quality, open source, change capture connectors for a variety of databases. If you are planning on streaming data from a relational database to Kafka, it is highly recommend using a Debezium change capture connector if one exists for your database. In addition, the Debezium documentation is one of the best we’ve seen— in addition to documenting the connectors themselves, it covers useful design patterns and use cases related to change data capture, especially in the context of microservices.

You can also refer sample connectors https://github.com/wushujames/kafka-mysql-connector

And the performance parameters will be purely driven from your current infrastructure setup. As you know, there are many variants of KAFKA Connectors (Source/Sink), and each behaves differently in the runtime.

We had a very complex event processing requirements as yours, sourcing CDC from Oracle and we had to the Golden Gate Connector and it is yet another flavour which may caters most of your requirements in terms of the Performance and Resiliency.

Refer : https://dzone.com/articles/creates-a-cdc-stream-from-oracle-database-to-kafka

Upvotes: 0

Related Questions