CODEWITHSUNDEEP
apache-kafka
Abhinav SInghvi
Abhinav SInghvi

Reputation: 411

Atomic message processing in kafka

I want to know the recommended way to process messages on kafka topic in atomic manner so for example let'a assume kafka producer is publishing multiple messages with key k1, k2, k3 and now I want to process them atomically and let my application know about these messages together. There can be some work around to achieve this so for example pass id and count along with all messages which need to be processed together so that client waits until it receives all messages part of the same group. Is there some other recommended way to solve such problems with kafka i.e ability to process a batch of messages atomically so that consistency could be maintained across the keys. Does kafka consumer provides such ability?

Upvotes: 1

Views: 1282

Answers (1)

Kavithakaran Kanapathippillai
Kavithakaran Kanapathippillai

Reputation: 8213

Aspects to address when producing

  • There is no atomicity that spans between producer and consumer in Kafka or any other message broker`.
  • So when producer send messages, you have to have some kind of co-relation id as part of the message, so consumers knows which messages belong to same group.
  • But knowing which messages belong together is not sufficient enough for the consumer, it should also know when to consider messages belong to a particular group has been collected completely so it can start processing a group. So far example, if it is fixed size groups, you don't need to send any group size as part of the message, otherwise either you need to add group size or something else in the produced message which signals to consumer the group is complete.
  • You should also produce the messages belong to the same group to the same partition.

Now you have basic needs satisfied, for rest of the work you can choose different paths.

  • For example, you can use something like camel-kafka and aggregator EIP to consume this topic and write to a different topic where each record is the whole group message and then you know you can consume from that topic atomically

Upvotes: 1

Related Questions