CODEWITHSUNDEEP
performanceapache-stormapache-kafka
paalaak
paalaak

Reputation: 183

Monitoring Kafka Spout with KafkaOffsetMonitoring tool

I am using the kafkaSpout that came with storm-0.9.2 distribution for my project. I want to monitor the throughput of this spout. I tried using the KafkaOffsetMonitoring, but it does not show any consumers reading from my topic.

I suspect this is because I have specified the root path in Zookeeper for the spout to store the consumer offsets. How will the kafkaOffsetMonitor know that where to look for data about my kafkaSpout instance?

Can someone explain exactly where does zookeeper store data about kafka topics and consumers? The zookeeper is a filesystem. So, how does it arrange data of different topics and their partitions? What is consumer groupid and how is it interpreted by zookeeper while storing consumer offset?

If anyone has ever used kafkaOffsetMonitor to monitor throughput of a kafkaSpout, please tell me how I can get the tool to find my spout?

Thanks a lot, Palak Shah

Upvotes: 2

Views: 2930

Answers (2)

Karthik Karuppaiya
Karthik Karuppaiya

Reputation: 31

Kafka-Spout maintains its offset in its own znode rather than under the znode where kafka stores the offsets for regular consumers. We had a similar need where we had to monitor the offsets of both the kafka-spout consumers and also regular kafka consumers, so we ended writing our own tool. You can get the tool from here:

https://github.com/Symantec/kafka-monitoring-tool

Upvotes: 1

mbaxi
mbaxi

Reputation: 1301

I have never used KafkaOffsetMonitor, but I can answer the other part.

zookeeper.connect is the property where you can specify the znode for Kafka; By default it keeps all data at '/'.

You can access the zookeeper filesystem using zkCli.sh, the zookeeper command line. You should look at /consumers and /brokers; following would give you the offset

get /consumers/my_test_group/offsets/my_topic/0

You can poll this offset continuously to know the rate of consumption at spout.

Upvotes: 0

Related Questions