CODEWITHSUNDEEP
javaapache-kafkaapache-kafka-connect
ADITYA SURVE
ADITYA SURVE

Reputation: 21

How to read from the _confluent-metrics topic in Java which is created by kafka

I want to get the value for sink-record-active-count for one of my sink connectors given here https://docs.confluent.io/current/connect/managing/monitoring.html#sink-task-metrics

I have all containers Running in docker desktop using docker compose file docker ps docker Ps

I have used confluent-metrics reporter for the task.

Also referring to https://docs.confluent.io/5.4.0/kafka/metrics-reporter.html and https://neo4j.com/docs/labs/neo4j-streams/current/examples/#_confluent_with_docker I have added the env variables to the kafka-container like this

 kafka-service:
    image: confluentinc/cp-enterprise-kafka:5.4.0
    container_name: kafka
    depends_on:
      - zookeeper
    links:
      - zookeeper
    ports:
      - 9092:9092
    expose:
      - "29092"
    environment:
      METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_REPORTER_TOPIC_CREATE: 'true'
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: http://kafka-service:29092
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-service:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100
    command:
      - bash
      - -c 
      - |
        echo '127.0.0.1 kafka-service' >> /etc/hosts
        /etc/confluent/docker/run
        sleep infinity

And in the kafka logs I am getting the message

INFO Created metrics reporter topic _confluent-metrics (io.confluent.metrics.reporter.ConfluentMetricsReporter)

I am not quiet sure how can I read from this topic in Java. Also will this topic have the required metrics? related to sink connector?

Thirdly on this page https://docs.confluent.io/current/connect/managing/monitoring.html#sink-task-metrics there is Mbeans given .. Also i am not sure how to use that . If it needs JMX too I had tried putting KAFKA_JMX_HOSTNAME = localhost and KAFKA_JMX_PORT: 9010 after following this https://rmoff.net/2018/09/17/accessing-kafka-docker-containers-jmx-from-host/ but I am not sure how to proceed ahead.

Upvotes: 2

Views: 1160

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191983

To my knowledge, ConfluentMetricsReporter is proprietary binary format, and cannot be read without the deserialziation libraries that are part of Control Center.

I suggest using Prometheus JMX Exporter + Grafana to have this data be visualized outside of a Kafka Consumer application or Control Center (such a setup is already offered by the Confluent Helm Charts)

Sidenote: Please don't edit any /etc/hosts files

Upvotes: 1

Related Questions