CODEWITHSUNDEEP
prometheuspromql
Mornor
Mornor

Reputation: 3783

Prometheus - Aggregate and relabel by regex

I currently have the following Promql query which allow me to query the memory used by each of my K8S pods:

sum(container_memory_working_set_bytes{image!="",name=~"^k8s_.*"}) by (pod_name)

The pod's name is followed by a hash defined by K8S:

weave-net-kxpxc
weave-net-jjkki
weave-net-asdkk

Which all belongs to the same app: weave-net

What I would like is to aggregate the memory of all pods which belongs to the same app.

So, the query would sum the memory of all weave-net pods and place the result in an app called weave. Such as the result would be: 

{pod_name="weave-net"}            10

instead of 

{pod_name="weave-net-kxpxc"}       5
{pod_name="weave-net-jjkki"}       3
{pod_name="weave-net-asdkk"}       2

Is it even possible to do so, and if yes, how ?

Upvotes: 29

Views: 22931

Answers (2)

James Mason
James Mason

Reputation: 4296

Iván's answer is spot on. label_replace doesn't work for range vectors (and it seems likely to stay that way), but there's a workaround using subqueries:

sum( 
  rate(
    label_replace(
      container_memory_working_set_bytes{image!="",name=~"^k8s_.*"}, 
      "pod_set", 
      "$1", 
      "pod_name", 
      "(.*)-.{5}"
    )[5m:] <-- it's the trailing colon here that makes it work
  )
) by (pod_set)

Upvotes: 4

Iv&#225;n Alegre
Iv&#225;n Alegre

Reputation: 1896

You can use label_replace

sum(label_replace(container_memory_working_set_bytes{image!="",name=~"^k8s_.*"}, "pod_set", "$1", "pod_name", "(.*)-.{5}")) by (pod_set)

You will be including a new label (pod_set) that matches the first group ($1) from matching the regex over the pod_name label. Then you sum over the new label pod_set.

[Edited]

Upvotes: 38

Related Questions