CODEWITHSUNDEEP
prometheusistioenvoyproxy
Ludovic C
Ludovic C

Reputation: 3065

Istio - How to add custom label-based metrics?

I am using istio 1.6.3

I would like to add a simple dimension to the metrics exported by istio to prometheus. More specifically, if my Pod has a label branch=master, I'd like to add a branch dimension with the master value to the istio_requests_total metric.

(I tried adding this label on the service level, without avail)

My goal is to then be able to query the metrics on prometheus, with

sum(rate(istio_requests_total[5m])) by (branch)

I read this piece of documentation: https://istio.io/latest/docs/tasks/observability/metrics/customize-metrics/

But it seems like getting the destination.labels["branch"], or getting any label at all is not supported (apart from the app or version labels, which are builtin destination_app and destination_version).

Help!

Upvotes: 0

Views: 1243

Answers (1)

Ludovic C
Ludovic C

Reputation: 3065

So just as a reference.

  1. Istio configures prometheus with a 'kubernetes-pods' job. At least while using the 'demo' profile. In this prometheus job config, there is a
relabel_configs: 
...
- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)

Which gets the pod labels.

  1. Enabling meshConfig.enablePrometheusMerge=true on the istio operator or whathever installation you are using will append the labels to the istio metrics.

Upvotes: 1

Related Questions