CODEWITHSUNDEEP
apache-kafka
Lucas Victor
Lucas Victor

Reputation: 59

Kafka and microservices - Architecture questions

In a Microservices based architecture, who writes to Kafka? services themselves or the Microservices databases? I've been thinking about this and see pros and cons to both approaches but leaning towards having database write to Kafka topics because

  1. Database and data in the Kafka topic won't go out of sync in case write to Kafka fails for whatever reason
  2. Application teams won't have to have one more step to worry about
  3. Applications can keep focusing on the core function rather than worrying about Kafka.

Thanks for your inputs

Upvotes: 1

Views: 341

Answers (2)

Gunnar
Gunnar

Reputation: 19020

As cricket_007 has been saying, databases typically cannot write to Apache Kafka themselves; instead, you'd need a change data capturing services such as Debezium in order to stream data changes from the database into Kafka (disclaimer: I'm the lead of Debezium).

Such an approach allows to ensure (eventual) consistency between a service's own database and Kafka messages sent to other services. On specific CDC application I'd recommend to look into is the outbox pattern. The idea there is to not capture changes to the service's actual business tables, but instead work with a separate "outbox table", into which the service writes specific messages meant for consumption by other services. CDC would then be used to sent these events from that table to Kafka.

This approach avoids exposing internal data structures to outside consumers while also avoiding the issues of "dual writes" which a service would suffer from when directly writing to its database and Kafka. In Debezium there's some means of built-in support for the outbox pattern via a message transformation that helps to route the events from the outbox table into event-type specific Kafka topics.

Upvotes: 2

OneCricketeer
OneCricketeer

Reputation: 191904

Not all services need a database, they just emit data (logs, metrics, sensors, etc)

So, the answer would be either.

Plus, I'm not sure what database directly can export to Kafka, so you'd have some other service like Debezium deployed which would be polling those CDC records off the database

Application developers still have to "worry" about how to deserialize their data, how many partitions are in the topic so they can scale out consumption, manage offsets, among other things

Upvotes: 0

Related Questions