CODEWITHSUNDEEP
prometheusgrafana
masoud
masoud

Reputation: 56549

Every Prometheus job as a data source in Grafana

We have a Prometheus, a Grafana service and multiple exporters. This is prometheus.yaml config:

scrape_configs: 
  - job_name: app1
    scrape_interval: 30s 
    scheme: http
    static_configs: 
      - targets: ['x.y.z.10:7380'] 

  - job_name: app2
    scrape_interval: 30s 
    scheme: http
    static_configs: 
      - targets: ['x.y.z.11:7380'] 

  - job_name: app3
    scrape_interval: 30s 
    scheme: http
    static_configs: 
      - targets: ['x.y.z.12:7380']

from Grafana's point of view, those scrapes will be accessed as a single data source. However, to make different dashboards for every application (app1, app2 and app3) we separated them by appending job filter {job="appX"} to queries/metrics.

This makes two problems, the first one is, we have to add {job="appX"} after every metric. The second one is, the list of metrics are populated of unrelated items from another jobs.

Is there any method in Prometheus or Grafana to make each job as separated as a data source? In which we don't have to add {job="appX"} to metrics. Let's say we don't have any access to another job's metrics in a specific dashboard.

Upvotes: 5

Views: 7757

Answers (3)

valyala
valyala

Reputation: 18094

I don't know a straightforward way to create per-job datasource in Grafana for metrics stored in Prometheus. But this can be easily done with VictoriaMetrics - just add extra_filters[]={job="appN"} query arg at custom query parameters input field per each datasource in Grafana. See the following additional information:

Upvotes: 1

Jens Baitinger
Jens Baitinger

Reputation: 2365

A good practice is to prefix your metric names with your application name as 'namespace'. e.g.:

appx_mycounter
appy_mycounter
appz_mycounter

This would enable you to ignore the job label completely and the typeahead in Grafana would work much better.

Upvotes: 1

Petar Nikolov
Petar Nikolov

Reputation: 319

No, there is no easy way to separate jobs into different data sources. The proper way imo you can handle it is to create a dashboard variable for the job_name which can be dynamically scraped, and have 1 dashboard which you can chose to show either job's metrics by adding {job="${variable}"} filter. If you don't want to add label filters after all the metrics you can use <relabel_config> to add a pre/sufix to all the metrics to make them for ex. app1_uptime https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

Upvotes: 3

Related Questions