CODEWITHSUNDEEP
apache-kafkaspring-kafkaapache-kafka-connect
Anoop Deshpande
Anoop Deshpande

Reputation: 582

What is the gain of using kafka-connect over traditional approach?

I have a use case where I need to send the data changes in relational database into a kafka-topic.

I'm able to write a simple JDBC program which executes set of queries for the changes in certain time period and write data into kafka-topic using KafkaTemplate (a wrapper provided by spring framework).

If I do the same using kafka-connect, which is to write a source connector. what benefits or overheads (if in case any) will I get?

Upvotes: 0

Views: 297

Answers (2)

Robin Moffatt
Robin Moffatt

Reputation: 32090

Today you want to ingest from a database using a set of queries from one database to a Kafka topic, and write some bespoke code to do that.

Tomorrow you want to use a second database, or you want to change the serialisation format of your data in Kafka, or you want to scale out your ingest or you want to have high availability. Or you want to add in the ability to stream data from Kafka to another target, to ingest data also from other places. And, manage it all centrally using a standardised configuration pattern expressed just in JSON. Oh, and you want it to be easily maintainable by someone else who doesn't have to read through code but can just use a common API of Apache Kafka (which is what Kafka Connect is).

If you manage to do all of this yourself—you've just reinvented Kafka Connect :)

I talk extensively about this in my Kafka Summit session: "From Zero to Hero with Kafka Connect" which you can find online here

Upvotes: 2

ppatierno
ppatierno

Reputation: 10065

The first thing is that you have "... to write a simple JDBC program ..." and take care of the logic of writing on both database and Kafka topic. Kafka Connect does that for you and your business application has to write to the database only. With Kafka Connect you have more than that like fail-over handling, parallelism, scaling, ... it's all out of box for you while you should take care of them when for example you write on the database but something fails and you are not able to write to Kafka topic and so on.

Upvotes: 2

Related Questions