Exporting spark executor jmx metrics for multiple executors running in same machine

Question

I am trying to scrape metrics for spark driver and executor using javaagent with below options. I have Prometheus in kubernetes cluster and I am running this spark application outside the kubernetes cluster.

spark.executor.extraJavaOptions=-javaagent:/opt/clkd/prometheus/jmx_prometheus_javaagent-0.3.1.jar=53700:executor_pattern.yaml

but I got below exception since both executors running on the same machine

Caused by: java.net.BindException: Address already in use .... 

I see many have posted the same question but I couldn't find the answer. Please let me know how can I resolve this issue.

Answer 1

I think that you need to switch from the pull-based monitoring, to push-based monitoring. For things, such as Spark jobs it makes more sense, as they aren't running all the time. For that you have some alternatives:

• Spark Prometheus Sink from Banzai Cloud as outlined in their blog post
• Setup GraphiteSink as described in the Spark documentation, and point it to the https://github.com/prometheus/graphite_exporter, and then scrape metrics from that exporter

---

Initial answer:

You can't have 2 processes listening on the same port, so just bind Prometheus from different jobs onto the different ports. Port is the number after the jmx_prometheus_javaagent-0.3.1.jar=, and before : character - in your case it's 53700. So you can use one port for one task, and another port (maybe 53701) for 2nd task...