CODEWITHSUNDEEP
node.jsamazon-web-servicesapache-kafkaaws-mskkafkajs
ppb
ppb

Reputation: 2633

service not consume message from all topic consumer group on AWS Kafka Cluster

I am very new to AWS and Kafka Cluster.

We have NodeJs service which connects to two different Kafka cluster. We have multiple instances of the consumer service and each instance consume a record from every topic but from last couple of weeks we found that service (consumers) not consuming message from every topics in list of subscribed topics, this set of topics are different every time. We have 5 different aws host for consumer service. When I run describe consumer group command I see some topics do not have any consumer attached. Every time this unassigned consumer topic list is different.

I apologies to updating question again, but while debugging the issue found this new issue.

Code

 KafkaProcessor.subscribetotopic = async (topic) => {
  const consumer = KafkaService.getConsumer(consumer);
  const message = {
   topic,
   fromBeginning: true,
  };
  await consumer.subscribe(message);
  logger.info(`Successfully subscribed to topic ${topic}`);
};

KafkaProcessor.subscribetotopiclist = async (topiclist) => {
 const promises = [];
 for (const topic of topiclist) {
  promises.push(KafkaProcessor.subscribetotopic(topic));
 }
 await Promise.all(promises);
 logger.info(`Successfully subscribed to topic ${topiclist.join()}`);
};

Upvotes: 0

Views: 502

Answers (1)

ChristDist
ChristDist

Reputation: 768

Better way to debug this issue is inspecting your consumer groups using ConsumerGroupCommand tool. This will give you the better glimpse of your consumer instances.

 bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group

--members --verbose: On top of the information reported by the "--members" options above, this option also provides the partitions assigned to each member.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members –verbose

Upvotes: 0

Related Questions