CODEWITHSUNDEEP
grafanaprometheus
and
and

Reputation: 2092

Tracking events with prometheus and grafana

There's an article "Tracking Every Release" which tells about displaying a vertical line on graphs for every code deployment. They are using Graphite. I would like to do something similar with Prometheus 2.2 and Grafana 5.1. More specifically I want to get an "application start" event displayed on a graph.

Grafana annotations seem to be the appropriate mechanism for this but I can't figure out what type of prometheus metric to use and how to query it.

Upvotes: 13

Views: 13409

Answers (3)

AussieDan
AussieDan

Reputation: 2176

The simplest way to do this is via the same basic approach as in the article, by having your deployment tool tell Grafana when it performs a deployment.

Grafana has a built-in system for storing annotations, which are displayed on graphs as vertical lines and can have text associated with them. It would be as simple as creating an API key in your Grafana instance and adding a curl call to your deploy script:

curl -H "Authorization: Bearer <apikey>" \
  http://grafana:3000/api/annotations \
  -H "Content-Type: application/json" \
  -d '{"text":"version 1.2.3 deployed",\
       "tags":["deploy","production"]}'

For more info on the available options check the documentation:

http://docs.grafana.org/http_api/annotations/

Once you have your deployments being added as annotations, you can display those on your dashboard by going to the annotations tab in the dashboard settings and adding a new annotation source: adding annotation source

Then the annotations is shown on the panels in your dashboard: panel showing annotation

Upvotes: 13

Sean Blanton
Sean Blanton

Reputation: 41

A note here as technology has evolved. We get deployment job state information in Prometheus metrics format scraped directly from the community edition of Hashicorp's Nomad and we view this information in Grafana.

In your case, you would just add an additional query to an existing panel to overlay job start events, which is equivalent to a new deployment for us. There are a lot of related metrics "out of the box," such as for a change in job version that can be considered as well. The main point is no additional work is required besides adding a query in Grafana.

enter image description here

Upvotes: 0

Alin S&#238;npălean
Alin S&#238;npălean

Reputation: 10134

You can get the same result purely from Prometheus metrics, no need to push anything into Grafana:

Annotation from Prometheus query

If you wanted to track all restarts your search expression could be something like:

changes(start_time_seconds{job="foo",env="prod"} > 0

Or something like this if you only wanted to track version changes (and you had some sort of info metric that provided the version):

alertmanager_build_info unless max_over_time(alertmanager_build_info[1d] offset 5m)

The latter expression should only produce an output for 5 minutes whenever a new alertmanager_build_info metric appears (i.e. one with different labels such as version). You can further tweak it to only produce an output when version changes, e.g. by aggregating away all other labels.

Upvotes: 7

Related Questions