CODEWITHSUNDEEP
grafanaprometheuspromql
Victor
Victor

Reputation: 151

Filter vector based new label introduced by `label_replace`

In Prometheus I'm trying to merge multiple http request lines into groups using label_replace.

http_requests_total{account_id="124",handler="AAAAAController"...}
http_requests_total{account_id="125",handler="BBBBBController"...}
http_requests_total{account_id="126",handler="CCCCCController"...}
http_requests_total{account_id="123",handler="XXXXXController"...}

The query I wrote is :

label_replace(http_requests_total, "class", "$1", "handler", "([a-zA-Z0-9]+)Controller.*") .

This works correctly and adds the class label to vector :"AAAA","BBBB" etc. At this point I would like to remove certain classes such as empty and BBBB.

How can I further filter the vector using {class~="BBBBB"} :

label_replace(http_requests_total, "class", "", "handler", "([a-zA-Z0-9]+)Controller.*"){class~="BBBBB"}

Prometheus shows an error when I try to do so.

Upvotes: 15

Views: 5013

Answers (6)

markalex
markalex

Reputation: 13360

As mentioned here already there is no easy and straightforward way of filtering this.

Some of the [simpler] filtering can be done with combination of set operators unless, and and or.

So, if you want to leave only metrics with label l1 equal to values v1 or v2 or empty value (missing label l1) you can use following:

<my_initial_query>
 and on(l1) (
   absent(_{l1="v1"}) or
   absent(_{l1="v2"}) or
   absent(_{})
 )

Similarly, to exclude metrics with label l1 equal to values v1 or v2 or empty value (missing label l1):

<my_initial_query>
 unless on(l1) (
   absent(_{l1="v1"}) or
   absent(_{l1="v2"}) or
   absent(_{})
 )

Here, absent(_{l1="v1"}) produces pseudo-metric with label l1="v1" (_ is simply used as a name of non-existing metric. You can use absent(non_existent{l1="v1"}) if it's easier).

Also, please note, that this trick is based on absent "tr[ying] to be smart about deriving labels" as official documentation states. This trick only works for label selectors using =. Something like l1=~"v1|v2|" wouldn't work.

Demo queries showing this principle in action can be found here.

Upvotes: 0

LoveXY
LoveXY

Reputation: 1

I had the same problems and this is my solution:

label_repalce({__name__=~"foo($lable)"}, 'lable', ...)

The label is which you will generate, you can set it on Grafana. This method works for me.

Upvotes: 0

DmitrySemenov
DmitrySemenov

Reputation: 10325

solution:

You can achieve it via joining two metrics, for example I need to group pods by nodes filtered by the name of the deployment that I pass as grafana variable

count by (node)
(
  label_replace(kube_pod_info{namespace="prod"}, "deployment", "$1", "pod", "(.*)-(.*)-(.*)$") 
  * on (deployment) group_left(job) 
       kube_deployment_labels{namespace="prod", deployment=~"prod-celery-beat-v1|prod-api-v1|prod-backend-v1"}
)

enter image description here

Upvotes: 1

valyala
valyala

Reputation: 17860

The filtering for this particular question can be done via the inner series selector before applying label_replace. For example, the following query returns only series with class=~"BBBBB":

label_replace(
  http_requests_total{handler=~"BBBBBController.*"},
  "class",
  "$1",
  "handler",
  "([a-zA-Z0-9]+)Controller.*"
)

P.S. While Prometheus doesn't provide the ability to post-filter time series by label values after they are selected by series selector, this can be done in VictoriaMetrics - Prometheus-like system I work on. See label_match and label_mismatch functions.

Upvotes: 5

Blithe
Blithe

Reputation: 21

I think the record rules feature might be helpful to you. Following is the steps for your reference

  1. create record rule. Prometheus allow us to set 2 type rules, one is alert rule another one is record rule. Record rule will create new metrics by the setting that you provide on Prometheus server setting. Here is the sample according your description.

groups:
- name: http_requests_total_new_merics
  rules:
  - record: http_requests_total_new
    expr: label_replace(http_requests_total, "class", "", "handler", "([a-zA-Z0-9]+)Controller.*")

then Prometheus will create new metrics named http_requests_total_new

  1. Using PromQL to query according your needs, e.g.
  • http_requests_total_new{class!="BBBBB"}

Upvotes: 0

澜塔1016
澜塔1016

Reputation: 746

I've noticed that there is a grafana tag under this question, so if you don't mind to use grafana to achive that goal, then there is another way.

By using Grafana... 7.x+ IIRC, you can use Transform to reprocess the raw result set.

For instance, I have a Query result like this:

enter image description here

Add a Filter data by values plugin. You can set Match any in it, and add a regex condition. Set field Class matches BBBBB or anything you want.

enter image description here

Upvotes: 1

Related Questions