CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
user377628
user377628

Reputation:

Is it possible to use Kafka Connect to mirror an RDBMS table to a Kafka Stream?

I know it's possible to push updates from a database to a Kafka stream using Kafka Connect. My question is, can I create a consumer to write changes from that same stream back into the table without creating an infinite loop?

I'm assuming if I create a consumer that writes updates into the database table, it would trigger Connect to push that update to the stream, etc. Is there a way around this so I can mirror a database table to a stream?

Upvotes: 0

Views: 465

Answers (2)

Robin Moffatt
Robin Moffatt

Reputation: 32110

You can stream from a Kafka topic to a database using the JDBC Sink connector for Kafka Connect.

You'd need to code in your business logic for avoiding an infinite replication loop into either the connectors or your consumer. For example:

  • JDBC Source connector uses a WHERE clause to only pull records with a flag set to indicate they are the original record
  • Custom Single Message Transform in the source connector to drop records with a flag set to indicate they are not the original record
  • Stream application (e.g. KSQL / Kafka Streams) processes the inbound stream of all database changes to filter out only those with a flag set to indicate they are the original record
    • Inefficient because then you're still streaming everything from the database

Upvotes: 2

Nick
Nick

Reputation: 348

Yes. It is possible to configure synchronisation/replication.

Upvotes: -1

Related Questions