How to draw separate plots for two instances in Prometheus?

Question

I have two instances running a job and sending gauge metrics to pushgateway.

For some reason Prometheus draws one plot for one job and two different instances. Here are the plots:

The question is what am I doing wrong and how to make Prometheus to draw separate continuous plots for these instances?

I'm using python's prometheus_client.

Answer 1

In Prometheus, a metric is identified by the name of the metric AND the set of labeled dimensions. Each time your instance label changes, it is another metric and thus another graph.

In order to merge the metric, you have to compute a resulting metric without the label that changes. Average is often a good candidate but doesn't matter if the metrics don't overlap:

avg(perf_test_concurrency) without(instance)