CODEWITHSUNDEEP
mysqlapache-kafkadebeziumchange-data-capture
Nipun
Nipun

Reputation: 4319

Getting near real time updates from normalised tables

My question is related to Change data capture and real time updates. I have a mysql database which has close to 250 tables whose change data is captured using debezium and sent to kafka on their respective topics.

These tables are highly normalised and I am interested in a denormalised view which is created by joining some of the tables. Now I need to know whenever there is a change in the table, I get to see that as a change in the denormalised topic in a near real time.

How can I do so. How does log compaction works? Do I need to log compact all the topics in order to get denormalised topic view?

What other possibilities I have here?

Upvotes: 4

Views: 158

Answers (1)

Ramachandran.A.G
Ramachandran.A.G

Reputation: 4948

This is not very straight forward to accomplish.

The following are some options you could try.

  1. The first option is using KSQL

https://debezium.io/blog/2018/05/24/querying-debezium-change-data-eEvents-with-ksql/

  1. The other option is to create aggregate views

https://debezium.io/blog/2018/09/20/materializing-aggregate-views-with-hibernate-and-debezium/

  1. Another option that we tried for a similar problem was to have streaming listener on Kafka and then have logic there to merge data into de-normal data structures that have been set up on another datastore

Upvotes: 2

Related Questions