CODEWITHSUNDEEP
sql-serverapache-kafkaapache-kafka-connectcdcdebezium
Nithya
Nithya

Reputation: 91

Kafka : Read from SQL server with change tracking enabled

I have been trying to load data from SQL server (with change tracking enabled) into Kafka, so that it can be consumed by one or many systems (reports, other DB's etc)

I have managed to configure the Kafka connect plugin for sql server(confluentinc/kafka-connect-cdc-mssql:1.0.0-preview) and i have also managed to start it on the kafka machine.

I have been looking for documentation (cannot find any) that helps answer the following questions

  1. How do i associate a kafka topic with this connection ?
  2. Based on the information i have found (on debezium forums) a topic would be created per individual table --> does it work the same way with the kafka sql server connector ?
  3. I have configured the connection in a distributed mode, we have kafka running on multiple servers, do we need to run the connection on every server ?
  4. Has anyone used Debezium with sql server change tracking and kafka ? the website for Debezium described the connection in the "alpha stages" and i was wondering if there were any active users.

P.S: I am also open to other options for loading real time data from sql server into Kafka (jdbc connection with a timestamp/numerical field is my backup option. Backup option as there are a few tables in my source database that do not contain such fields[changes are not and cannot be tracked with numerical/timestamp fields]).

Upvotes: 3

Views: 1783

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191874

1 & 2 -- How do i associate a kafka topic with this connection

I would believe it's per table, but you might be able to use a RegexRouter Connect transform to merge multiple tables into a single topic.

3 -- configured the connection in a distributed mode, we have kafka running on multiple servers, do we need to run the connection on every server

Kafka Connect should run outside of your Kafka servers. It is independently scalable.

4 -- Debezium with sql server change tracking

I have not. Probably a better question to ask to the Debezium mailing lists or JIRA tickets working on these features.

Upvotes: 0

Related Questions