Rationale behind Prometheus Labeldrop Action

Question

I understand that in order to restrict sending unnecessary metrics to the storage systems, we can use metrics_relabel_configs and action "labeldrop" to drop certain labels.

All the examples I see, people are dropping labels as if they already knew all the labels associated with the metrics.

While using exporters like node_exporters, process_exporter and kubestatemetrics for kubernetes, there are just too many metrics present. How do you decide which one to keep or drop ?

Answer 1

labeldrop only drops a label attached to a metrics, which is good as Brian says for pruning unnecessary info labels. Once that action is completed the metric name and labelset must still be unique so it's not mixed up with another metric.

In order to drop a whole metric, the drop action is the one to use, which uses a regular expression to target a trait of a metric - including the __name__ label - to decide if the whole metric should be removed.

The quickest way to get a look at the available labels from any one target is to have a look at the Prometheus UI in the targets section e.g. prometheus:9090/targets. On that page you can see each target for a scrape job, and the labels that are being attached to the stored metrics. If you hover over those labels, you can also see what the labels were before any rewrites were applied to them.

Some relabel actions are automatic, e.g. __address__ is renamed to instance, and the labeldrop action is applied automatically to all labels with a __double_underscore__ at the start of the name after all relabel rules have been applied. That means there's no need to drop any rules which start with a double underscore.

As far as choosing which metrics to drop entirely, that depends on your needs, whether you have any recording rules in place, and at what point you're dropping them.

• if you have a metric which counts requests received by an application, and you run the application in multiple containers, it might make sense to have a recording rule in place to aggregate all those metrics into one overall count, and drop all the original metrics.

• It might also make sense to keep all those metrics in your short-term local storage, but use write_relabel_configs to drop them before they get sent to long-term remote storage.

Also worth noting, there's another action called keep which does the reverse of drop, dropping everything except metrics which match. A good use for this action is to use annotations applied to pods to ensure Prometheus only gets the correct metrics for a specific job. There's an example of that in my blog post and video, where the pods to be scraped for metrics for a specific job are annotated prometheus.io/scrape: "true". That annotation is passed through service discovery as a label called __meta_prometheus_pod_annotation_prometheus_io_scrape and the keep rule is based on whether the value of the label is true.

Answer 2

The labeldrop action exists as cAdvisor used to add a lot of varying and unneeded labels to every pod metric, and it allows you to remove them. This is the sort of use case labeldrop and labelkeep are for, when a target is exposing labels on metrics that should really have gone on an info metric.