CODEWITHSUNDEEP
jmxprometheusjmx-exporter
Umapathy
Umapathy

Reputation: 772

exporting custom metrics through prometheus jmx exporter

I can see jvm, process, java etc metrics on a /metrics endpoint when using the jmx_exporter but not the custom metrics I have exposed through JMX.
When I remove the agent and add a jolokia agent, I can see those custom metrics without problem on Jolokia's exposed endpoint in JSON format. That proves it exists.

jmx exporter agent config

rules:
- pattern: ".*"

I have scoured through stackoverflow and google groups. Many people have raised similar issues but I can't see a solution. Any help on why this is not working as expected.

Upvotes: 2

Views: 5999

Answers (2)

Martin Naughton
Martin Naughton

Reputation: 1556

I followed this tutorial and i was able to https://reachmnadeem.wordpress.com/2020/12/06/monitoring-jmx-enabled-java-applications-with-prometheus/

i used this config just as a starting point for the jmx export to show its information on its metrics page.

---
jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:5555/jmxrmi
startDelaySeconds: 0
ssl: false
rules:
- pattern: 'com.company.monitoring:*'
  name: consumer
  value: 3
  valueFactor: 0.001
  labels: {}
  help: "Cassandra metric"
  cache: false
  type: GAUGE
  attrNameSnakeCase: false

Upvotes: 0

FK82
FK82

Reputation: 5075

You need to add a whitelist entry with a pattern for the object names of your custom JMX Beans.

Example: Assume domain foo, then adding the following to your JMXExporter configuration YAML at top level

whitelist: ["foo:*"]

will print all the metrics for the beans in the foo domain.

Sidenote: I found it kind of tricky to work with pattern entries in the rule set, so I'm gonna add my findings in case you run in this follow up problem.

So, the pattern string for the pattern entries in the rules section must apply to a specific normalized representation for bean metrics (see documentation):

domain<beanpropertyName1=beanPropertyValue1, beanpropertyName2=beanPropertyValue2, ...><key1, key2, ...>attrName: value

If you provide a pattern entry in a rule item, this is used in JMXCollector.java (line 174-175) to create a regex pattern as follows

if (yamlRule.containsKey("pattern")) {
    rule.pattern = Pattern.compile("^.*(?:" + (String)yamlRule.get("pattern") + ").*$");
}

JMXCollector uses this to check if your rule set applies to a given bean (or otherwise apply the default exposition formatter).

JMXExporter will aid you a little in constructing a pattern matcher by providing the normalized format in the HELP ... line if you let it use the default formatter (by not adding an entry to the rules section). If you do a run with no rules, you can copy the normalized bean metric representation and use this to define a matching pattern.

Upvotes: 2

Related Questions