CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectdebezium
Stefano
Stefano

Reputation: 83

Kafka Connect: a sink connector for writing multiple tables from one topic

I'd like to use Kafka Connect to detect changes on a Postgres DB via CDC for a group of tables, and push them as messages in one single topic, with the key as the logic key of the main table.

This will give the consumer to consume the data changes in the right order to apply them to a destination DB.

Are there Source and Sink connectors allowing me to achieve this goal?

I'm using Debezium CDC Source Connector for Postgres... which I can configure to route all the messages for all the tables into one single topic.

But then I'm not able to find a Sink connector capable to consume the messages, and write to the right table depending on the schema of the message.

Upvotes: 0

Views: 2203

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191864

There'd be no property for a specific sink connector, if that's what you're looking for

You can use Single Message Transforms to Extract parts of the record to set the outgoing topic name from a single sink connector topic

Example : https://docs.confluent.io/platform/current/connect/transforms/extracttopic.html

Upvotes: 0

Related Questions