CODEWITHSUNDEEP
prometheus
roie
roie

Reputation: 865

How to execute multiple queries in one call in Prometheus

I'm running prometheus inside kubernetes cluster.

I need to send queries to Prometheus every minute, to gather information of many metrics from many containers. There are too match queries, so I must combine them.

I know how I can ask Prometheus for one metric information on multiple containers: my_metric{container_name=~"frontend|backend|db"} , but I haven't found a way to ask Prometheus for multiple metric information in one query.

I'm looking for the equivalent to 'union' in sql queries.

Upvotes: 31

Views: 44251

Answers (3)

valyala
valyala

Reputation: 18066

Prometheus provides or operator, which can be used for combining results from multiple queries in a single API call. For example, the following query returns results for q1, q2 and q3:

q1 or q2 or q3

The or operator in PromQL has a quirk though - if some time series returned by q1, q2 and q3 have the same set of labels (excluding metric name aka __name__ label), then only the first encountered time series is returned. For example, the following query returns only process_cpu_seconds_total time series, and doesn't return process_resident_memory_bytes time series, since they have the same set of labels except of metric name:

process_cpu_seconds_total or process_resident_memory_bytes

This can be fixed by copying the metric name into another label. For example, the following query copies the metric name into metric_name label, so the or result contains time series from both process_cpu_seconds_total and process_resident_memory_bytes sides:

label_join(process_cpu_seconds_total, "metric_name", "", "__name__")
  or
label_join(process_resident_memory_bytes, "metric_name", "", "__name__")

P.S. There is a simpler solution for obtaining results from multiple queries in one call if using an alternative Prometheus-like monitoring system I work on - VictoriaMetrics - it provides union function for this task. For example, the following query returns time series for both process_cpu_seconds_total and process_resident_memory_bytes queries:

union(
  process_cpu_seconds_total,
  process_resident_memory_bytes,
)

Upvotes: 4

roie
roie

Reputation: 865

I found here this solution: {__name__=~"metricA|metricB|metricC",container_name=~"frontend|backend|db"}.

Upvotes: 50

brian-brazil
brian-brazil

Reputation: 34172

You can use the or operator, however this does not generalise as it ignores metric names. I'd suggest making multiple queries to the API.

Upvotes: 7

Related Questions