CODEWITHSUNDEEP
apache-kafkadata-ingestion
Sushant Choudhary
Sushant Choudhary

Reputation: 31

Is there any api or tool for getting ingestion ratio in a kafka topic

How can I find the ingestion ratio? i.e., no of events/bytes ingested every second/minute interval

Upvotes: 0

Views: 717

Answers (3)

sivann
sivann

Reputation: 2141

You can monitor real time message rates on topics/consumer groups using https://github.com/sivann/kafkatop, it's a TUI.

Upvotes: 0

Mickael Maison
Mickael Maison

Reputation: 26895

As mentioned in the other answer, Kafka exports metrics via JMX that cover bytes in/out rates as well as a number of other interesting data points. For the full details, see the Monitoring section in the docs.

It's recommended to collect and graph any of these metrics in you are planning to run a Kafka cluster.

Now if you just want to quickly get these values while developing, you have a couple of other options:

  • jconsole: This graphical tool comes with the JVM and can display all the JMX metrics.

  • kafka.tools.JmxTool: This CLI tool comes with Kafka and can print the value of JMX metrics. For example, to get the broker ingestion rate, run:

    ./bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec

For these to work, you must enable the JVM to expose JMX on a port.

Upvotes: 1

mythic
mythic

Reputation: 625

You can use a jmx exporter in your kafka setup and send the scraped metrics to prometheus(time series db).

The metrics in prometheus can then be visualised in grafana. You will get topic level metrics there where you can see the ingestion per second/minute for every topic

Upvotes: 0

Related Questions