CODEWITHSUNDEEP
prometheus
sebastian
sebastian

Reputation: 2358

Finding the source of a prometheus metric

How do you find the source of a prometheus metric? For example, the "up" metric. How do you go about finding the exporter that is making that metric visible?

If you go to the "targets" view in prometheus UI, you can get a list of all the endpoints that are scraped. however some of them use https and if you curl them you get unauthorized.

Is there another way to go about this?

Upvotes: 2

Views: 2702

Answers (1)

YwH
YwH

Reputation: 1150

Just for sharing some of my experience, not necessarily the best practice:

  1. How do you find the source of a prometheus metric?

    Look at the job label of the metric, find the corresponding target in the "Targets" view of prometheus UI, where you should find the metric source. If not found there, the job label is defined in the metric itself. Find target jobs with honor_labels: true config in the "Configuration" view of prometheus UI. Your metric should stay is those targets.

  2. however some of them use https and if you curl them you get unauthorized.

    You could curl the target endpoints in the prometheus container or one pod container nearby using the same serviceaccount, turn off curl's verification of the certificate with -k, as follows:

    TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -H "Authorization: Bearer $TOKEN" https://xx.xx.xx.xx:10250/metrics/cadvisor -k

Upvotes: 3

Related Questions