CODEWITHSUNDEEP
jmxapache-kafkajmxtrans
bert2002
bert2002

Reputation: 67

jmxtrans and kafka not working

I am trying to get the Kafka metrics (version 0.8.2) to my grafana server. Unfortunately I can only get java.lang metrics, but no kafka metrics. Connecting with jmxtrans and jconsoel is now problem and I can see the MBeans of Kafka.

Configuration of jmxtrans:

{
  "servers" : [ {
    "port" : "9393",
    "host" : "localhost",
    "queries" : [  {
      "obj" : "kafka.network:type=RequestMetrics,name=LocalTimeMs,request=ConsumerMetadata",
      "attr" : [ "Count" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
        "settings" : {
          "port" : 2003,
          "host" : "localhost"
        }
      } ]
    }, {
      "obj" : "kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=FetchConsumer",
      "attr" : [ "Count" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
        "settings" : {
          "port" : 2003,
          "host" : "localhost"
        }
      } ]
    }, {
      "obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=OffsetFetch",
      "attr" : [ "Count" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
        "settings" : {
          "port" : 2003,
          "host" : "localhost"
        }
      } ]
    }, {
      "obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Offsets",
      "attr" : [ "Count" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
        "settings" : {
          "port" : 2003,
          "host" : "localhost"
        }
      } ]
    }
   ],
"numQueryThreads" : 2
  } ]
}

Log of jmxtrans:

15:52:11.564 [ServerScheduler_Worker-10] DEBUG org.quartz.core.JobRunShell - Calling execute on job ServerJob.localhost:9393-1443714611563-0428747962
15:52:11.564 [ServerScheduler_Worker-10] DEBUG c.googlecode.jmxtrans.jobs.ServerJob - +++++ Started server job: Server [host=localhost, port=9393, url=null, cronExpression=null, numQueryThreads=2]
15:52:11.569 [ServerScheduler_Worker-10] DEBUG com.googlecode.jmxtrans.jmx.JmxUtils - ----- Creating 4 query threads
15:52:11.638 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Executing queryName [kafka.network:name=RequestQueueTimeMs,request=FetchConsumer,type=RequestMetrics] from query [Query [obj=kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=FetchConsumer, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]]
15:52:11.638 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Finished running outputWriters for query: Query [obj=kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=FetchConsumer, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]
15:52:11.640 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Executing queryName [kafka.network:name=TotalTimeMs,request=OffsetFetch,type=RequestMetrics] from query [Query [obj=kafka.network:type=RequestMetrics,name=TotalTimeMs,request=OffsetFetch, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]]
15:52:11.640 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Finished running outputWriters for query: Query [obj=kafka.network:type=RequestMetrics,name=TotalTimeMs,request=OffsetFetch, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]
15:52:11.641 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Executing queryName [kafka.network:name=TotalTimeMs,request=Offsets,type=RequestMetrics] from query [Query [obj=kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Offsets, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]]
15:52:11.641 [pool-60-thread-2] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Finished running outputWriters for query: Query [obj=kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Offsets, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]
15:52:11.642 [pool-60-thread-1] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Executing queryName [kafka.network:name=LocalTimeMs,request=ConsumerMetadata,type=RequestMetrics] from query [Query [obj=kafka.network:type=RequestMetrics,name=LocalTimeMs,request=ConsumerMetadata, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]]
15:52:11.642 [pool-60-thread-1] DEBUG c.g.jmxtrans.jmx.JmxQueryProcessor - Finished running outputWriters for query: Query [obj=kafka.network:type=RequestMetrics,name=LocalTimeMs,request=ConsumerMetadata, useObjDomainAsKey:false, resultAlias=null, attr=[Count]]
15:52:11.643 [ServerScheduler_Worker-10] DEBUG c.googlecode.jmxtrans.jobs.ServerJob - +++++ Finished server job: Server [host=localhost, port=9393, url=null, cronExpression=null, numQueryThreads=2]

For me it looks like that jmxtrans can connect without problem and tries to get the data, but does not get any data at all from the kafka metrics.

JMX Options of Kafka:

-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9393 -Djava.rmi.server.hostname=example.com -Dcom.sun.management.jmxremote.rmi.port=9011

Any idea why the Kafka metrics are not available?

Thanks.

Upvotes: 2

Views: 1178

Answers (1)

Eric Broda
Eric Broda

Reputation: 7231

Some versions of Kafka had quoted JMX elements (you could see the attributes having quotes in the jconsole). This has/is changing for future versions of Kafka. Check out the this post (How to monitor Kafka broker using jmxtrans?) for details, but if you use quoted strings it should work (see example below:)

"obj": "\"kafka.server\":type=\"BrokerTopicMetrics\",name \"AllTopicsLogBytesAppendedPerSec\"", "attr": [ "Count" ]

Note that the "obj" value requires additional quotes. When I tried this, JMXTRANS provided valid output after putting the correct (quoted) values in the obj string...

Upvotes: 1

Related Questions