CODEWITHSUNDEEP
apache-kafkaprometheusapache-kafka-connectjmx-exporter
floating_hammer
floating_hammer

Reputation: 439

Is there a way to configure kafka-connect jmx metrics to be captured using a jmx_exporter/prometheus?

I'm setting up monitoring for Kafka connect in our Kafka ecosystem. I have enabled JMX exporter for kafka brokers and is working fine. Now I am trying to enable JMX exporter for kafka connect. However, it is a bit unclear where to start.

I can only modify connect-distributed.sh to enable the change. Any pointers would be a great addition.

kafka-run-class.sh was modified to enable jmx_exporter to emit jmx metrics on http://<host>:9304/metrics

I expect kafka-connect to emit metrics on http://<host>:19000/metrics once the jmx_exporter has been enabled.

Upvotes: 1

Views: 6591

Answers (2)

Iskuskov Alexander
Iskuskov Alexander

Reputation: 4375

Modifying a script in the bin directory is highly unrecommended. When upgrading Kafka to the next version, extracting the new binaries would override the changes made in the script.

The preferred way should be to set the environment variable KAFKA_JMX_OPTS outside the script:

export KAFKA_JMX_OPTS="-javaagent:/opt/kafka/libs/jmx_prometheus_javaagent-0.12.0.jar=127.0.0.1:10902:/etc/kafka-connect/jmx_exporter.yaml"

If the var is set before starting Kafka via the script it will use the var instead of the default values defined in /bin/kafka-server-start.sh

This answer is inspered by How do I set the Java options for Kafka?

Upvotes: 7

floating_hammer
floating_hammer

Reputation: 439

Digging around I have found a solution to the issue. Java agent can be added to the connect-distributed.sh and starts to emit the metrics nicely.

So enable jmx_exporter on Kafka Connect framework

  1. Open connect-distributed.sh
  2. Modify the last line which calls the kafka-run-class.sh
exec $(dirname $0)/kafka-run-class.sh $EXTRA_ARGS org.apache.kafka.connect.cli.ConnectDistributed "$@"

to the following

exec $(dirname $0)/kafka-run-class.sh $EXTRA_ARGS -javaagent:/path/to/jmx_prometheus_javaagent-0.11.0.jar=9408:/path/to/config/file/prometheus.yml org.apache.kafka.connect.cli.ConnectDistributed "$@"

Upvotes: 0

Related Questions