CODEWITHSUNDEEP
apache-kafka
Vinod Jayachandran
Vinod Jayachandran

Reputation: 3898

How to communicate to kafka consumer that Kafka producer is complete?

I have a scenario where a Producer produces finite number of messages and a set of consumers belonging to a consumer group consume and process the message.

What's the ideal way to communicate to all the consumers that producer has completed queuing the finite set of messages ? I was thinking of the following approaches

  1. Broadcast a COMPLETE message to all consumers
  2. Producer updates the status in a external db/file

Is there any implicit feature within Kafka to achieve it ?

Upvotes: 3

Views: 1327

Answers (1)

Sönke Liebau
Sönke Liebau

Reputation: 1973

Kafka doesn't offer a feature to accomplish this as it is mostly geared towards streaming use cases.

I think your ideas are headed in the right direction, you need to somehow enable the producer to communicate directly with the consumers, how that is achieved can of course be a vast number of possibilities:

  • update external file/db
  • enable consumers to receive a rest call
  • write a metaevent to the Kafka topic which is ignored by consumers but signals the end of the stream
  • ...

There are a few stumbling stones to keep in mind for all of these approaches though.

Partitions

If your topic has several partitions you cannot simply send one done message to the topic, as this will be assigned to only one of the partitions. Since ordering is only guaranteed per partition it is entirely possible, that the consumer reads the done message before it processes messages from other topics that should have been read as well and stops processing. For this case you'd need to ensure that you send a done event to every single partition and your consumer needs to be aware of this concept as well and keep reading until it got a done message for every subscribed partition.

Duplicates

Same deal if you choose to go with an external trigger, you'll probably want to tell your consumer how many events to receive before it is done. There are cases where the consumer might read messages twice though, if it crashes before committing the read offset for example, in this case simply transmitting a number of messages sent value is not enough, as these duplicates would mean that you are ignoring messages at the end. Just like with the first idea, you would want to transmit the last offset that your producer sent for every partition to your consumer, so that it knows the offset to read until for every partition.

Upvotes: 5

Related Questions