CODEWITHSUNDEEP
apache-sparkmetricsexecutorscodahale-metrics
eje
eje

Reputation: 945

How to enable ExecutorAllocationManagerSource metrics in Apache Spark?

The documentation on enabling general metrics in Apache Spark is kind of thin:

Within an instance, a "source" specifies a particular set of grouped metrics. there are two kinds of sources:

Spark internal sources, like MasterSource, WorkerSource, etc, which will collect a Spark component's internal state. Each instance is paired with a Spark source that is added automatically.

Common sources, like JvmSource, which will collect low level state. These can be added through configuration options and are then loaded using reflection.

All the examples are of the form:

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource

However, none of the plausible-seeming variations on this allowed me to publish the metrics generated in ExecutorAllocationManagerSource

The class isn't unit tested, and I can't locate any other documentation or examples.

Upvotes: 1

Views: 242

Answers (1)

eje
eje

Reputation: 945

In fact, these metrics are published without any special configuration for ExecutorAllocationManagerSource. However, they only manifest if the relevant code paths are active. In this case, that means enabling dynamic executor allocation.

My cluster had been mistakenly configured without dynamic executor allocation. When that is turned on, these metrics are published under driver metrics, without any special configuration.

Upvotes: 1

Related Questions