CODEWITHSUNDEEP
apache-flink
holly
holly

Reputation: 1

Prometheus doesn't have metrics from taskmanager if flink job started

I operate flink 1.15.2 on Kubernetes and set metric configuration for Flink Cluster as below

# metrics
metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter

The problem is that prometheus doesn't get metrics from taskmanager if the flink job has started. If I stopped the job, then I could see the metrics however some metrics are empty.

  1. I tried to reduce CPU usage but still no metric from taskmanager
  2. I tried to increase task slot, still no metric
  3. It happens to both Intel and ARM node
  4. I tried to change flink config as below, metircs were collocted for a moment(several seconds) and disappeared again
# metrics
metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
  1. I tried to change flink config as below, but still no metric
kafkaSourceBuilder.setProperty("register.consumer.metrics", "false");
var producerProperties = new Properties();
producerProperties.setProperty("register.producer.metrics", "false");
producerSinkBuilder.setKafkaProducerConfig(producerProperties);
  1. If I try to start job on flink 1.15.3, metircs were collocted
  2. If I try to start job on flink 1.16.0, Prometheus doesn't have any metric from flink at all

Upvotes: 0

Views: 691

Answers (1)

Martijn Visser
Martijn Visser

Reputation: 2108

As mentioned in the release notes of Flink 1.16, configuring reporters by their class has been deprecated. See https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.16/#flink-27206httpsissuesapacheorgjirabrowseflink-27206 for details.

There are also some known issues with metrics reporting in 1.16.0; please upgrade to Flink 1.16.1.

Upvotes: 2

Related Questions