CODEWITHSUNDEEP
goapache-kafkakafka-consumer-apikafka-producer-apiconfluent-platform
user923499
user923499

Reputation: 343

Create multiple consumers for same topic in kafka

I am newbie and can see one example with one consumer in below github repository, but any ideas how to create multiple consumers for same topic in go lang?

https://github.com/confluentinc/confluent-kafka-go/tree/master/examples

Any consumer factory (to generate N consumers) available in confluent-kafka to read same topic (with partitions)?

Upvotes: 1

Views: 8258

Answers (1)

Nishu Tayal
Nishu Tayal

Reputation: 20810

There is an example in the Confluent github repo :

https://github.com/confluentinc/confluent-kafka-go/blob/master/examples/consumer_example/consumer_example.go

If you want to create multiple consumers for the same topic, there are two scenarios :

1.Create each consumer with different group id. 

c1, err := kafka.NewConsumer(&kafka.ConfigMap{
        "bootstrap.servers":    broker,
        "group.id":             group1,
        "session.timeout.ms":   6000,
        "default.topic.config": kafka.ConfigMap{"auto.offset.reset": "earliest"}})

c2, err := kafka.NewConsumer(&kafka.ConfigMap{
        "bootstrap.servers":    broker,
        "group.id":             group2,
        "session.timeout.ms":   6000,
        "default.topic.config": kafka.ConfigMap{"auto.offset.reset": "earliest"}})
  1. If you want multiple consumers for the same topic but with the same group id, it will start consuming from single partition. i.e. A topic contains 3 partitions and you create 3 consumers with same group id, each consumer will consume from one partition

Upvotes: 1

Related Questions