CODEWITHSUNDEEP
apache-kafkaprometheusbitnamiservicemonitorkafka-exporter
gstackoverflow
gstackoverflow

Reputation: 37034

How to configure kafka chart to expose both: kafka-metrics and kafka-jmx-metrics to prometheus?

We use kafka chart (version 17) as a dependency of our chart. And for I try to configure monitoring of that kafka.

values.yaml:

kafka:
  metrics:
    kafka:
      enabled: true
    jmx:
      enabled: true
    serviceMonitor:
      enabled: true
      labels:
        my.custom.label/service-monitor: "1.0"
   ...

Then I deploy my application and go to prometheus targets and I see following:

enter image description here

I am concerned about the failed request.

What I see ?

There are 2 services to monitor:

and both of those services are requested by 2 paths:

But kafka-exporter doesn't respond on /. Is there way to configure kafka chart to work without failures if there are 2 exporters are enabled ?

P.S.

Services definition:

spec:
  endpoints:
    - path: /
      port: http-metrics
  namespaceSelector:
    matchNames:
      - my-ns
  selector:
    matchLabels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/instance: dev-my-app
      app.kubernetes.io/name: my-app-kafka

spec:
  endpoints:
    - path: /metrics
      port: http-metrics
  namespaceSelector:
    matchNames:
      - my-ns
  selector:
    matchLabels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/instance: dev-my-app
      app.kubernetes.io/name: my-app-kafka

Upvotes: 0

Views: 2360

Answers (2)

gstackoverflow
gstackoverflow

Reputation: 37034

Eventually I finished with following solution:

I use jmx exporter from kafka chart:

values.yaml

kafka:
  metrics:
    kafka:
      enabled: false 
    jmx:
      enabled: true
    serviceMonitor:
      enabled:true
      labels:
       my.super.company/service-monitor: "1.0"

  enabled: true

and registered kafka exporter as separated dependency:

chart.yaml:

  - name: prometheus-kafka-exporter
    version: 1.6.0
    repository: https://prometheus-community.github.io/helm-charts

and configured it manually:

values.yaml

prometheus-kafka-exporter:
  kafkaServer:
    - my-kafka-server:9092
  prometheus:
    serviceMonitor:
      enabled: true
      namespace: ""
      additionalLabels:
        my.super.company/service-monitor: "1.0"

Upvotes: 1

OneCricketeer
OneCricketeer

Reputation: 191728

kafka-exporter doesn't respond on /.

That's because web.telemetry-path defaults to /metrics

https://github.com/danielqsj/kafka_exporter/blob/master/kafka_exporter.go#L700

Your ServiceMonitors are mixed up

  • the one that ends in kafka-jmx-metrics should have container=jmx-exporter labels
  • one that ends with kafka-metrics should have container=kafka-exporter

Currently, you've shown there is one of each container in the opposite ServiceMonitors. I suspect this is because your label selectors are the exact same, but for different paths.

If you only have one Kafka broker, you should only have 2 targets. One JMX Exporter (for each broker) and one Kafka exporter.
So, two brokers would have 3 targets: 2 JMX Exporters and one Kafka exporter (since it is for the whole cluster).

Upvotes: 1

Related Questions