CODEWITHSUNDEEP
apache-flinkprometheusmetrics
sarvad123
sarvad123

Reputation: 21

Apache Flink showing custom metrics in UI, but prometheus metrics reporter not scraping them

I am working on sending custom app metrics to prometheus via the Prometheus Flink Metrics Reporter. The metrics are correctly created since I am able to accurately see them in the flink dashboard. I configured the prometheus metrics reporter similar to found here. When I curl to the prometheus endpoint (curl http://localhost:9090/api/v1/metrics), I am only able to see the cluster metrics and not the custom metrics I am creating. I suspect this issue has to do with how I configured the Prometheus Flink Metrics Reporter since when I try to visit http://localhost:9090, there is no UI and just a list of the cluster metrics mentioned above.

flink job code to create metrics(visible in Flink UI):

this.anomalyCounter = getRuntimeContext.getMetricGroup.addGroup("metric1").counter("counter")

flink-conf.yaml:

metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.port: 9090

promethus.yml:

scrape_configs:
  - job_name: 'flink'
    static_configs:
      - targets: ['localhost:9090']

Is there anything I am missing in the configuration? Why are my cluster metrics reaching prometheus and not my custom ones?

Upvotes: 2

Views: 1318

Answers (2)

john gorman
john gorman

Reputation: 11

I've seen similar issues based on a bug in the 1.13.6 Flink we were using. The reporter was blowing up and thus you got no custom metrics. This has been fixed in 1.16 version we are using now and we can view both custom and rocksdb metrics. For what it's worth the 1.13.6 version had lots of issues that apparently made the Flink UI pretty useless for data reporting. 1.16 is much more stable and reports things quite well.

Upvotes: 1

Niko
Niko

Reputation: 682

Hi @sarvad123 probably (depending on your Flink version) you should add flink-metrics-prometheus-{version}.jar in the /lib folder.

Upvotes: -1

Related Questions