CODEWITHSUNDEEP
pythonapache-kafkakafka-python
arturo_bandini
arturo_bandini

Reputation: 21

Multiple consumers with same group id in Python

Does anybody know how to run multiple consumers with same group id in Python? I've tried following

a = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
b = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
c = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
a.subscribe([topic_to_read])
b.subscribe([topic_to_read])
c.subscribe([topic_to_read]) 
running = True
while running:
    msg1 = a.poll(timeout=timeout)
    msg2 = b.poll(timeout=timeout)
    msg3 = c.poll(timeout=timeout)

But this is not working. So I've tried using multiprocessing lib but I am not able to make it work.

Upvotes: 0

Views: 2854

Answers (2)

subzero
subzero

Reputation: 366

Check how many partitions you have for that topic. The number of consumers within a group id should not exceed the number of partitions of the topic the group is consuming from. Else the extra consumers will remain idle. ALSO CHECK IF YOU ARE ASSIGNING DIFFERENT CLIENID/CONSUMERID FOR EACH OF THE CONSUMERS.

Upvotes: 0

bmarkham
bmarkham

Reputation: 1670

A group ID is a unique ID to each consumer. If you are subscribing to a same topic with multiple consumers, you must have different group ID's or else only one of those consumers will get the messages.

Upvotes: 1

Related Questions