CODEWITHSUNDEEP
apache-flinkgrafanaprometheusflink-streaming
Jack
Jack

Reputation: 1941

How to monitor Flink Backpressure in Grafana with Prometheus metrics

Flink Web UI has a brilliant backpressure section. But I can not see any metrics, given by Prometheus reporter, which could be used to detect backpressure in the same way for a Grafana dashboard.

enter image description here Is there some way to get the same metrics outside of the Flink Web UI? Using the metrics described here https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html. Or even having a prometheus scraper for scraping the web api?

Upvotes: 2

Views: 1533

Answers (1)

David Anderson
David Anderson

Reputation: 43707

The back pressure monitoring that appears in the Flink dashboard isn't using the metrics system, so those values aren't available via a MetricsReporter. But you can access this info via the REST api at

/jobs/:jobid/vertices/:vertexid/backpressure

While this back pressure detection mechanism is useful, it does have its limitations. It works by calling Thread.getStackTrace(), which is expensive, and some operators (such as AsyncFunction) do critical activities in threads that aren't being sampled.

Another way to investigate back pressure is to set this configuration option in flink-conf.yaml

taskmanager.network.detailed-metrics: true

and then you can look at the metrics measuring inbound/outbound network queue lengths.

Upvotes: 7

Related Questions