CODEWITHSUNDEEP
kubernetesapache-kafkakafka-consumer-api
jannath hussain
jannath hussain

Reputation: 23

Only one Kafka consumer is getting data

There is one Kubernetes property called replica, suppose we have 3 replicas(replica-0, replica-1, replica-2) and we are producing 100 messages in Kafka, Only one replica say (replica-0) is invoked at a time for consuming all the 100 messages, but the expected solution is all the 100 messages should be divided among 3 replicas(replica-0, replica-1, replica-2) instead keeping the load on one replica for consuming the messages

Upvotes: 1

Views: 502

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 192023

You should clarify "invoked". Consumers poll; they are not externally "invoked"/started other than calling poll() method.

Consumers configured with the same group.id cannot subscribe to overlapping partitions. If you sent 100 records to one partition, then only one consumer will read that. If you want all applications to read a subset of the data, then you first will need a Kafka topic with at least 3 partitions (the default auto-created topics only have one), then send data amongst those.

Kubernetes has nothing to do with the Kafka consumer subscription.

Upvotes: 1

Related Questions