Using Grafana templates for Prometheus data sources

Question

Grafana 4.3.1, Prometheus 1.8.0

We have 3 Prometheus data sources which represent different data centers. I have created a dashboard in Prometheus using the "Mixed" data source option to allow a view across the data centers by default. I would like to include a "template" which essentially provides a drop down that will allow the selection of data source 1..3 or ALL. Is there anyway to include something like an "IF" statement in the promql which will essentially hide certain panel metrics if this drop down is set to certain values?

Grafana repeat row/panel - I realize this is an option but it would introduce new panels and rows for each data center(data source) selected. I am trying to find out if I can keep the panels static and update the contents based on the selected value.

OBJECTIVE - Have a dashboard which can either view across all data sources or isolate to 1 with a simple selection of a drop down (Template).

Answer 1

I was unable to find a perfect solution which would avoid having to create multiple queries (Metrics). The solution I have found is to include label filter in the underlying promql and a new custom Grafana template (drop-down) for our data centers.

The old way had 3 queries each pointed at different data source:

avg(node_load1{pod="$pod",package="$package"})

The new way has the same three, but a new label filter which will remove the metric unless the target data center is selected or "all" (Grafana "Include all option") is selected.

avg(node_load1{pod="$pod",package="$package",location=~"($location|all)"})

This adds some overhead to the dashboard, but achieves the objective of the dashboard.

Answer 2

Instead of using "Query" option in templating, you can go for "datasource" which will give you all the datasources in the drop down list.

But for this, your datasources should already be created.

I have 4 datasources and I used the same kind of templating.