Kafka + using prometheus have confilcts with kafka cli commands

Question

We are using Prometheus for some time and really enjoying it.

Few words about What is jmx-exporter

jmx-exporter is a program that reads JMX data from JVM based applications (e.g. Java and Scala) and exposes it via HTTP in a simple text format that Prometheus understand and can scrape.

So let’s get started with my issue …

We configured the kafka with jmx exporter as the following

export KAFKA_OPTS="-javaagent:/home/jmx_prometheus_javaagent-0.11.0.jar=7071:/home/kafka-2_0_0.yml"

this configuration set in ambari under kakfa config

after setting the configuration we restart all 3 kafka brokers

we Check that jmx-exporter HTTP server is listening:

netstat -tlnp | grep 7071
tcp6       0      0 :::7071                 :::*                    LISTEN      63872/java

And scrape the metrics!

curl -s localhost:7071 | grep -i kafka | head
# HELP kafka_log_logcleanermanager_max_dirty_percent Attribute exposed for management (kafka.log<type=LogCleanerManager, name=max-dirty-percent><>Value)
# TYPE kafka_log_logcleanermanager_max_dirty_percent gauge
kafka_log_logcleanermanager_max_dirty_percent 0.0

until now every thing is cool

but when we start to use kafka commands , for example to print the list of topics we get:

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh –zookeeper $zookeeper_server:2181 –list

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at sun.net.httpserver.ServerImpl.bind(ServerImpl.java:133)
        at sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
        at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
        at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:49)
        ... 6 more
FATAL ERROR in native method: processing of -javaagent failed

Note – before we add the following line:

export KAFKA_OPTS="-javaagent:/home/jmx_prometheus_javaagent-0.11.0.jar=7071:/home/kafka-2_0_0.yml "

everything was ok

so any advice – how to start to solve this problem from this point ?

more reference -

https://alex.dzyoba.com/blog/jmx-exporter

https://medium.com/@mousavi310/monitor-apache-kafka-using-grafana-and-prometheus-873c7a0005e2

Answer 1

as per your errors its because either your zookeeper or kafka is already running with the ports.

1. stop all the process for kafka/zookeepers
2. add the EXTRA_VAR with the javaagent and define your jmx_exporter with a any different from the kafka ports and assign the *.yml (i.e broker.yml) file.
3. now when you restart your kafka/zookeeper it would add your jmx_exporter and *.yml (i.e broker.yml) .
4. monitor your kafka/zookeeper are started with ps -ef | grep kafka . you would check that your jmx_exporter and broker.yml are included in this

Answer 2

Of all the solutions I found, it helped me to change the KAFKA_OPTS variable to EXTRA_ARGS. But I still don't understand the cause of the conflict. Kafka runs on one port and javaagent on another. But nevertheless, when creating topics, an error appears. Javaagent I am using to pass Prometheus metrics

Answer 3

Kafka CLI scripts are making use of the same env vars that the broker is using(in particullar KAFKA_OPTS and JMX_PORT). That's why they are conflicting. I was using confluent container(confluentinc/cp-server:5.5.1) with:

JMX_PORT: 9991
KAFKA_OPTS: "-javaagent:/usr/local/bin/jmx_prometheus_javaagent-0.13.0.jar=7071:/etc/jmx-exporter/kafka-2_0_0.yml"
volumes:
  - ./jmx_exporter/kafka-2_0_0.yml:/etc/jmx-exporter/kafka-2_0_0.yml
  - ./jmx_exporter/jmx_prometheus_javaagent-0.13.0.jar:/usr/local/bin/jmx_prometheus_javaagent-0.13.0.jar

A lot of threads are pointing to JMX_PORT only but that didn't help me. I had to unset both env vars from inside the container before I was able to run CLI commands. It works as follows:

unset JMX_PORT
unset KAFKA_OPTS
kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic your_topic

It will not mangle your broker because those vars are set per session.

Answer 4

Your Kafka instance is listening for prom scrapes on port 7071 which seems fine. When you issue Kafka commands, it looks like the javaagent you have added to the KAFKA_OPTS is also trying to listen on port 7071. Not sure why you need the javaagent there but try a different port there.

Answer 5

Ref : https://github.com/wurstmeister/kafka-docker/wiki#why-do-kafka-tools-fail-when-jmx-is-enabled

Tools such as kafka-topics.sh and kafka-console-producer.sh fail when JMX is enabled. This is caused because of the JMX_PORT environment variable. The Kafka helper script /opt/kafka/bin/kafka-run-class.sh will try to invoke the required command in a new JVM with JMX bound to the specified port. As the broker JVM that is already running in the container has this port bound, the process fails and exits with error.

Solution would be to either prefix your command with JMX_PORT= or unset the environment variable, i.e. unset JMX_PORT