CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
Abhimanyu
Abhimanyu

Reputation: 2740

Monitoring Kafka Clusters , Producers and Consumers

I am new to Kafka and I have three nodes Kafka cluster with multiple consumers and producers.I want to monitor the Kafka cluster on the basis of three parameters :

1.Topic wise monitoring - to get incoming rate of topic , lag in a topic, number of active consumers attached with the topic, number of partitions
2.Consumer based monitoring - the topic from which it is consuming,consumption speed. a number of messages consumed.
3.Message-Based Monitoring - topic name in which message is produced,and status of message i.e whether it's processed or pending.

I have tried various monitoring tools like (KafkaOffsetMonitor,Burrow) but none of them are matching my use case .So can anyone suggest how to setup this kind of monitoring system?.Are there APIs available through which I can get the above-mentioned details

Upvotes: 3

Views: 3583

Answers (2)

Luciano Afranllie
Luciano Afranllie

Reputation: 4253

We use two things.

Take a look at this great blog about monitoring Kafka, most of the tools and articles about monitoring Kafka are summarized there.

Upvotes: 1

amethystic
amethystic

Reputation: 7079

you could fire up JConsole and monitor the whole Kafka cluster with JMX configured. See https://kafka.apache.org/documentation#monitoring for details.

Upvotes: 2

Related Questions