CODEWITHSUNDEEP
apache-sparkprometheusdatabricksspark-streamingspark-structured-streaming
Shane
Shane

Reputation: 626

How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus

I have been reading this article - https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch duration into prometheus.

I was able to get host/infra metrics like memory, disk etc via below API.

https://eastus-c3.databricks.net/driver-proxy-api/o/<org-id>/<cluster-id>/40001/metrics/executors/prometheus

I couldnt find any apis or references to get the streaming metrics, processing info etc. Any help on how to get those streaming UI metrics to prometheus ?

Spark configs being set on cluster:

spark.ui.prometheus.enabled true
spark.sql.streaming.metricsEnabled true

Here is the prometheus config file:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'prometheus'
    scheme: https
    scrape_interval: 5s
    static_configs:
      - targets: ['eastus-c3.azuredatabricks.net']
    metrics_path: '/driver-proxy-api/o/<orgid>/<clusterid>/40001/metrics/executors/prometheus'
    basic_auth:
      username: 'token'
      password: 'user gen token'

enter image description here

Upvotes: 0

Views: 1107

Answers (1)

Christopher Grant
Christopher Grant

Reputation: 101

Streaming metrics are emitted from the driver, not the executors.

Try /driver-proxy-api/o/<orgid>/<clusterid>/40001/metrics/prometheus.

For driver metrics, you'll need to enable the PrometheusServlet. You can do this on Databricks by attaching an init script like:

#!/bin/bash
cat << EOF > /databricks/spark/conf/metrics.properties
*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus
master.sink.prometheusServlet.path=/metrics/master/prometheus
applications.sink.prometheusServlet.path=/metrics/applications/prometheus
EOF

And don't forget to name your streaming queries (Python, Scala), this helps identify the query and differentiate queries when executing multiple streaming queries on the same cluster.

Upvotes: 1

Related Questions