CODEWITHSUNDEEP
springspring-bootapache-kafkakafka-consumer-apispring-kafka
Fabian Lang
Fabian Lang

Reputation: 78

Report Metrics from Kafka to Actuator

I'm trying to get some metrics (client lag, ...) from kafka to provide it for consumption by prometheus.

My approach would be to write a simple springboot application which exposes the metrics for prometheus. I understand that kafka provides metrics to all its consumers via the interface MetricsReporter.

So I implemented a class which should do exactly that:

public class MonitoringIntegration implements MetricsReporter {

    @Override
    public void init(List<KafkaMetric> list) {
        System.out.println("init");
        for (KafkaMetric kafkaMetric : list) {
            System.out.println(kafkaMetric.metricName());
            System.out.println(kafkaMetric.metricValue());
        }
    }

    @Override
    public void metricChange(KafkaMetric kafkaMetric) {
        System.out.println("Metric Change");
        System.out.println(kafkaMetric.metricName());
        System.out.println(kafkaMetric.metricValue());
    }

    @Override
    public void metricRemoval(KafkaMetric kafkaMetric) {
        System.out.println("Removal");
        System.out.println(kafkaMetric.metricName());
        System.out.println(kafkaMetric.metricValue());
    }

    @Override
    public void close() {
        System.out.println("close");
    }

    @Override
    public void configure(Map<String, ?> map) {
        System.out.println("Configuring");
        System.out.println(map);
    }
}

I registered this class with a bean:

@Configuration
public class MetricConfiguration {

    @Bean
    public ProducerFactory<?, ?> kafkaProducerFactory(KafkaProperties properties) {
        Map<String, Object> producerProperties = properties.buildProducerProperties();
        producerProperties.put(CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
                MonitoringIntegration.class.getName());
        return new DefaultKafkaProducerFactory<>(producerProperties);
    }

    @Bean
    public ConsumerFactory<?, ?> kafkaConsumerFactory(KafkaProperties properties) {
        Map<String, Object> consumererProperties = properties.buildConsumerProperties();
        consumererProperties.put(CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
                MonitoringIntegration.class.getName());
        return new DefaultKafkaConsumerFactory<>(consumererProperties);
    }
}

When I start the application some metrics will be printed out to cmd, but they have all default values (0.0, infinite, ..) and they will only be provided once after the application started.

Why am I not getting the metrics? What did I do wrong?

Cheers,

Fabian

Upvotes: 2

Views: 7563

Answers (2)

Luiz Toscano
Luiz Toscano

Reputation: 21

The consumer metrics support are only available on spring boot 2.1+ versions.

Auto-configuration Support For New Metrics Metrics coverage has been improved to include:

Hibernate metrics

Spring Framework’s WebClient

Kafka consumer metrics

Log4j2 metrics

Jetty server thread pool metrics

Server-side Jersey HTTP request metrics

https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-2.1-Release-Notes#auto-configuration-support-for-new-metrics

I recommend you to upgrade to newer versions. But if you really need to use Spring Boot prior versions, you can check my kafka metrics micrometer implementation at:

https://github.com/luiztoscano/spring-boot-kmetrics

Upvotes: 0

Shankar
Shankar

Reputation: 2825

Spring Kafka already exposes Kafka metrics as a JMX metrics. You dont need to update/send the metrics to Prometheus. Prometheus server will automatically read from your application's "/prometheus" endpoint. Enable Spring Actuator with Prometheus in your Spring project and configure the Prometheus server to read from it.

Here is a great example using Spring Boot - https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/

MetricsReporter is not used to "report" metric values as they change. Check the docs. (For some reason I cant find the latest API).

https://archive.apache.org/dist/kafka/0.8.2-beta/java-doc/org/apache/kafka/common/metrics/MetricsReporter.html

A plugin interface to allow things to listen as new metrics are created so they can be reported.

metricChange() method will only be called when a metric is changed. This is the reason you see the first few outputs during application startup, because the metrics were created.

Upvotes: 2

Related Questions