CODEWITHSUNDEEP
apache-kafkajmx
Dan M
Dan M

Reputation: 1272

Kafka Monitoring: request latencies from JMX

We want to monitor Kafka and have two specific requirements: use headless tools and store performance metrics in a CSV file. Following Gwen Shapira series [1] I am leaning towards request latencies and kafka.tools.JmxTool to start with.

Setup: Kafka 0.11, exposed JMX, headless metric collection tools

Q: what JMX beans provide metrics as presented on [2], likely per Broker: “request queue”, “request local”, “response remote”, “response queue”, “response send”?

[1] slides
https://www.slideshare.net/ConfluentInc/metrics-are-not-enough-monitoring-apache-kafka-and-streaming-applications/

[2] desired Kafka metrics

desired Kafka metrics

Upvotes: 0

Views: 1813

Answers (3)

Dave Canton
Dave Canton

Reputation: 178

Here they are in addition to some description of what they mean. I hope it helps.

Request Queue: time that request waits at the request queue

kafka.network:type=RequestChannel,name=RequestQueueSizeMs

Request Local: time processing the request at the leader

kafka.network:type=RequestMetrics,name=LocalTimeMs,request=Produce

Response Remote: time waiting for the followers to process the request

kafka.network:type=RequestMetrics,name=RemoteTimeMs,request=Produce

Response Queue: time that the request waits at the response queue

kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=Produce

Response Send: time to send the response

kafka.network:type=RequestMetrics,name=ResponseSendTimeMs

Upvotes: 0

Dan M
Dan M

Reputation: 1272

After some exploring, here is the full set to open Kafka JMX at port 3999 and gather the request metrics:

1 Open Kafka JMX port at 3999:

update bin/kafka-run-class.sh:

# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=0.0.0.0 -Djava.net.preferIPv4Stack=true"
fi

# JMX port to use
if [  $JMX_PORT ]; then
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=${JMX_PORT} -Dcom.sun.management.jmxremote.rmi.port=${JMX_PORT} "
fi

update bin/kafka-server-start.sh:

if [ -z "$JMX_PORT" ]; then
  export JMX_PORT=3999
fi

2 bash script to gather Kafka metrics and publish them to the journalctl:

#!/bin/bash

PIPE=/tmp/kafka-monitoring-temp.out
mkfifo $PIPE

# Start logging to journal
systemd-cat -t 'kafka-monitoring' < $PIPE &
sleep_pid=$(
        sleep 9999d > $PIPE &  # keep pipe open
        echo $!                # but allow us to close it later...
)

arrayName=( "kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=Fetch"
            "kafka.network:type=RequestMetrics,name=LocalTimeMs,request=Fetch"
            "kafka.network:type=RequestMetrics,name=RemoteTimeMs,request=Fetch"
            "kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=Fetch"
            "kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=Fetch"
            )

for name in "${arrayName[@]}"; do
        timeout 1s /ust/lib/kafka/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name "${name}" --jmx-url service:jmx:rmi:///jndi/rmi://127.0.0.1:3999/jmxrmi --reporting-interval 1100 | tee $PIPE
done

kill $sleep_pid
rm $PIPE

Upvotes: 3

Alex Ott
Alex Ott

Reputation: 87069

There is a good chapter on Monitoring in the "Kafka: The Definitive Guide" (pdf is freely available from Confluent's site). The book shows following Request-related metrics:

Request metrics

Upvotes: 0

Related Questions