CODEWITHSUNDEEP
prometheusprometheus-operator
Neeraj Krishna
Neeraj Krishna

Reputation: 1615

How to deal with "prometheus" and "prometheus_replica" labels?

In a hierarchical federated setup of prometheus with a Pull model for the metrics, I see "prometheus" and "prometheus_replica" labels in the metrics that's captured. The system is monitoring a StatefulSet deployment of Kubernetes.

When querying or alerting I see duplicate data included due to these labels, i.e I see a metric with these labels and also without these. Effectively causing wrong counts and alerts.

I see "prometheus" and "prometheus_replica" labels used in the queries on the prometheus that pulls metrics from federated endpoint.

I use ServiceMonitor with Prometheus operator on every kube cluster. All the metrics is federated to a single different Prometheus where this problem is seen.

Is there any documentation on how these labels get generated? Are those metrics to be treated duplicate or ignored?

Upvotes: 10

Views: 8968

Answers (4)

Umut Gerçek
Umut Gerçek

Reputation: 1056

If you use kube-prometheus-stack helm chart, there are options you can set:

These sets labels' values to "" under the hood.

Can check these values from UI YOUR_PROM_URL/config look at global.external_labels

Upvotes: 2

saranicole
saranicole

Reputation: 2453

For those using kube-prometheus-stack I found this very helpful - https://github.com/prometheus-operator/prometheus-operator/issues/455#issuecomment-1367809401 .

Use replicaExternalLabelName: "__replica__" and it will add a dynamic external label based on pod name. I also have Mimir installed and it references the external labels as:

config: |
    limits:
      ha_cluster_label: cluster
      ha_replica_label: __replica__

Upvotes: 0

Shay Rybak
Shay Rybak

Reputation: 1293

I've run into this issue as well, just adding for anyone getting to this comment the configuration options that affect this according to the documentation here https://github.com/prometheus-operator/prometheus-operator/blob/ca400fdc3edd0af0df896a338eca270e115b74d7/Documentation/api.md#prometheusspec . link to the code here https://github.com/prometheus-operator/prometheus-operator/blob/ca400fdc3edd0af0df896a338eca270e115b74d7/pkg/prometheus/promcfg.go#L95-L132

replicaExternalLabelName: Name of Prometheus external label used to denote replica name. Defaults to the value of prometheus_replica. External label will not be added when value is set to empty string ("").
prometheusExternalLabelName: Name of Prometheus external label used to denote Prometheus instance name. Defaults to the value of prometheus. External label will not be added when value is set to empty string ("").

So if you want to remove these duplicates, just set those options to empty strings in the Prometheus custom resource in your cluster.

Upvotes: 7

Neeraj Krishna
Neeraj Krishna

Reputation: 1615

I finally found these labels coming from the prometheus operator. It was added for an requirement that's unwritten in any documents. I see it doesn't work in 0.17 version. Its works in 0.23 version of operator.

Upvotes: 1

Related Questions