CODEWITHSUNDEEP
apache-kafkamicroservices
felix.specht
felix.specht

Reputation: 21

Microservices with Apache Kafka software architecture / patterns

We want to use the microservice pattern and Apache Kafka as an event driven message stream.
We have different use cases that we want to implement with Kafka, but we dont know how to do this with Kafka. Maybe we have a misconception there or Kafka isnt the right choice for us.

  1. problem: Service A (Producer) emits an event to the Kafka broker (a message will be sent to a certain topic). Now we have, lets say, 5 consumer services that listen to that event. All the services do different things, lets say one stores data in a database, one calculates something, etc. All the consumers should be called when the event is emitted, so i set them up to listen to the same topic but with different groups.
Service/Consumer B -> @KafkaListener(topics = "EventName", groupId = "GROUP1")`
Service/Consumer C -> @KafkaListener(topics = "EventName", groupId = "GROUP2")
Service/Consumer D -> @KafkaListener(topics = "EventName", groupId = "GROUP3")
    ...

I dont know if this is the correct way to achieve what we want to achieve.

Next we want service A to wait for all 5 consumers to finish their task, before further processing. Is that even possible or is it a misconception on my side? The next thing we want is that certain events should only be processed once (exactly once), but i think this is possible with disabling auto commits and handling the partition offset manual. I did not try this yet, and maybe i did mess things up there :-) but what ive read, this is possible. (Please correct me if it is not possible)

  1. problem: Data handling. Service A generates an data object, that holds most of data for a user session. At some point Service A emits an event and the 5 consumers will trigger and process something and then return/store their result. We want, for various reasons, to save these results in the same data object that Service A generated. First the producer could pass this data object as a parameter to the consumers, so they can do their calculations etc.

But how to save/return their result? They cant write in the passed data object, because of data consistency problems. Should they return their result and Service A stores the result in the data object?

Follow up question: Shouldnt we pass the data object at all and maybe the consumer should ask for the data with an extra event?

Maybe we should not use Kafka, maybe we should changes our patterns/architecture completely

Let me sum up what i want to achieve and what not:

Upvotes: 0

Views: 480

Answers (1)

Levi Ramsey
Levi Ramsey

Reputation: 20591

  1. Multiple consumer groups is the canonical way to get the same message to multiple consuming services.

If the initial event you're publishing has a correlation ID (something unique to that event), then each of the 5 consumers can publish an event to a Kafka topic saying that that consumer processed the event.

In general, exactly-once processing of a message is not possible. There's a very limited exception to this in Kafka (basically, if the processing consists only of generating a message and producing that message back to Kafka: any processing beyond this voids the exactly-once promise), but you'll almost certainly have to choose between at-most-once and at-least-once (you can get fairly close to exactly-once with at-least-once and a consumer which can remember what it's done).

  1. It's possible to implement such a thing (I've done it). The CQRS pattern will help here. Service A is the write-side for this data and publishes events to Kafka. It also subscribes to the topics published to by your 5 downstream services (which are in some sense read-sides). The events from those services (which are, for this purpose, "event with correlation ID XYZ has been processed (and maybe here are the results?)") become commands to service A which performs the appropriate updates.

Upvotes: 0

Related Questions