CODEWITHSUNDEEP
pythonprometheusgrafana
Pravin Kumar
Pravin Kumar

Reputation: 157

How to add an extra label to 1000s of prometheus queries in grafana dashboard programmatically?

I have almost 1000 prometheus queries configured in my grafana dashboard, and I need to add one more label to all the queries to get the correct metric, since this label was added recently to differentiate the metrics based on region. I know what the query is, but adding the new label manually in all 1000 queries seems a tedious and repetitive work. Is there a way to programmatically add a label to an existing query string (by using some parsing library in any programming language or regex pattern of queries)?

For eg, if I have a query:

sum(increase(django_http_responses_total_by_status_view_method_total{view="some_api_function", status="200"}[5m]))

I need to add one more label to it:

sum(increase(django_http_responses_total_by_status_view_method_total{view="some_api_function", status="200", region="us-east-1"}[5m]))

Adding it manually for lesser number of queries is not an issue, but for 1000s of such queries, it becomes difficult. I was even able to parse the grafana dashboard json and get all the queries in it using this snippet:

import json

with open("grafana-dashboad.json", "r") as f:
    dashboard_data = json.load(f)

panels = dashboard_data["panels"]

queries = []

for panel in panels:
    title = panel.get("title")
    # print(f"Title: {title}")
    targets = panel.get("targets", [])
    for target in targets:
        query = target["expr"]
        queries.append(query)
    inner_panels = panel.get("panels", [])
    for inner_panel in inner_panels:
        inner_targets = inner_panel.get("targets", [])
        for inner_target in inner_targets:
            query = inner_target["expr"]
            queries.append(query)
for query in queries:
    print(query)
print(len(queries))

But I am not sure how to proceed further to add the extra label in the query programmatically. Is there any best way or library to get this done automated, rather than updating the queries manually one-by-one? Can the metric somehow be parsed into a python object, so that the label can be updated? If not python, I am also fine with using any other language, as long as it makes the work easier.

Upvotes: 0

Views: 1040

Answers (1)

Pravin Kumar
Pravin Kumar

Reputation: 157

I was able to parse the promql query string as python objects, and add my own label to it using this python library: https://github.com/pabluk/pygments-promql

from pygments import highlight, lex, format
from pygments.formatters import NullFormatter
from pygments.token import Name, Operator, Literal, Punctuation
from pygments_promql import PromQLLexer

query = 'sum(increase(django_http_responses_total_by_status_view_method_total{view="some_api_function", status="200"}[5m]))'
# print(highlight(query, PromQLLexer(), NullFormatter()))
promql_lex = list(lex(query, PromQLLexer()))

label_start_index = None
for index, token in enumerate(promql_lex):
    if list(token)[0] == Name.Label and list(token)[1] == "view":
        # print("First label", token, type(list(token)[0]))
        label_start_index = index
        break
print(format(lex(query, PromQLLexer()), NullFormatter()))

new_label = [(Name.Label, "region"), (Operator, '='), (Punctuation, '"'), (Literal.String, "us-east-1"), (Punctuation, '"'), (Punctuation, ',')]
# print(format(new_label, NullFormatter()))
# print(label_start_index)
promql_lex[label_start_index:label_start_index] = new_label
# print(promql_lex)
print(format(promql_lex, NullFormatter()))

I have made this code as a function, and used it in the grafana dashboard parsing snippet that I added in my question to update all the queries in the dashboard json file.

Upvotes: 1

Related Questions