CODEWITHSUNDEEP
mysqlapache-kafkadebezium
Sorana Pop
Sorana Pop

Reputation: 3

How to capture batch events with Debezium and send them to Kafka?

I am building a bi-directional data replication app between MySQL and MongoDB using Spring Boot and Kafka. I have added the required dtos, services, repositories (using JPA), document classes, entity classes and so on. I am using the outbox pattern approach (https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html) in order to be able to capture every change that occurs in my tables (in a nutshell: there is an "Outbox" table that contains the operation type, table and payload of the change that occurs, so this way everytime a change happens somewhere, be it MySQL or Mongo, it gets added in the Outbox table and captured by Debezium, if configured so).

I want to be able to send batch messages to Kafka, so more than 1 entry when an update operation occurs for example. Should I just incorporate the data inside the "payload" column? Is there a better approach? Right now I am using the saveAll method provided by JPA when saving more than 1 events in the Outbox table, but I'm not really sure how or if it's possible to configure Debezium to listen to more than 1 change....

Thanks in advance!

Upvotes: 0

Views: 643

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191864

Kafka records should only be single events, but Kafka producer already batches requests; thus the producer config batch.size

If you want to aggregate data within Kafka, run a Kafka Streams Topology in your Spring app.

Upvotes: 0

Related Questions