CODEWITHSUNDEEP
pythonapache-kafkaconfluent-kafka-python
Ibrahim Khan
Ibrahim Khan

Reputation: 202

Consuming messages from Kafka with different group IDs using confluent_kafka

I am using python and confluent_kafka

I am building a Queue Management for Kafka where we can view the pending(uncommitted) messages of each topic, delete topic and purge topic. I am facing the following problems.

If I run consumer1 and consumer2 simultaneously only one consumer will start consuming and another just keep on waiting and hence cause heavy loading time in the frontend.

If I assign different group Id for each it works but, the messages committed by consumer1 are still readable by consumer2.

Example: If I have pushed 100 messages and say consumer1 consumed 80 messages and when I try to consume from consumer2 it should consume only remaining 20, but it is consuming all 100 including the messages committed by consumer1.

How can I avoid that or solve?

Upvotes: 1

Views: 790

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

Unclear what you mean by uncommitted. Any message in a topic has been committed by a producer.

From the consumer perspective, this isn't possible. Active Kafka consumers in the same group cannot be assigned the same partitions

More specifically, how would "consumer2" know when/if "consumer1" was "done consuming 80 records" without consumer1 becoming inactive?

If you have an idle consumer with only two consumers in the same group, sounds like you only have one partition... If you want both to be active at the same time, you'll need multiple partitions, but that won't help with any "visualizations" unless you persist your consumed data in some central location. At which point, Kafka Connect might be a better solution than Python.

If you want to view consumer lag (how far behind a consumer is processing), then there are other tools to do this, such as Burrow with its REST API. Otherwise, you need to use the get_watermark_offsets() function to find the topic's offsets and compare to the current polled record offset

Upvotes: 2

Related Questions