Instrumenting a Java Application With Prometheus

Question

I am trying to monitor multiple metrics in a java application using the prometheus java client library but I am having difficulties monitoring more than one metric at a time. If I register and scrape only one metric everything seems to work fine and I can see the metric using the prometheus application but if I try to expose and monitor more than one metric then only one of the metrics will be visible.

So If I have an application with the instrumented class as below

package com.telemetryserver.Instrumentation;

import io.prometheus.client.*;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

public class Instrumented_Class extends HttpServlet
{
    private static Instrumented_Class _instance = null;

    public static final Gauge metric_1 = Gauge.build().name("metric_1").help("metric_1").register();
    //public static final Gauge metric_2 = Gauge.build().name("metric_2").help("metric_2").register();

    public static Instrumented_Class getInstance()
    {
        if (_instance == null)
            _instance = new Instrumented_Class();

        return _instance;
    }

    //Getters and Setters

    public static void setMetric_1(double val) { metric_1.set(val); }

    public static double getMetric_1() { return metric_1.get(); }

    //public static void setMetric_2(double val) { metric_2.set(val); }

    //public static double getMetric_2() { return metric_2.get(); }

    @Override
    protected void doGet(final HttpServletRequest req, final HttpServletResponse resp) throws ServletException, IOException
    {

        double metric_1 = Instrumented_Class.getMetric_1();
        Instrumented_Class.setMetric_1(metric_1 + 1);

        //double metric_2 = Instrumented_Class.getMetric_2();
        //Instrumented_Class.setMetric_2(metric_2 + 2);

        resp.getWriter().println("Hello from Instrumented_Class!!!," +
                " metric_1 = " + metric_1
                //+ " metric_2 = " + metric_2
        );
    }
}

and the main class as

package com.telemetryserver.client;


import com.telemetryserver.Instrumentation.*;
import io.prometheus.client.exporter.MetricsServlet;
import io.prometheus.client.hotspot.DefaultExports;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;

public class TelemetryApp
{
    public static void main(String[] args)
    {
        startTestServer(2018);
    }


    public static void startTestServer(int port)
    {
        try
        {
            Server server = new Server(port);
            ServletContextHandler context = new ServletContextHandler();
            context.setContextPath("/");
            server.setHandler(context);

            //Expose our Instrumented servlet.
            context.addServlet(new ServletHolder(Instrumented_Class.getInstance()), "/");

            //Prometheus Metrics Servlet
            context.addServlet(new ServletHolder(new MetricsServlet()), "/metrics");

            // Add metrics about CPU, JVM memory etc.
            //DefaultExports.initialize();

            // Start the webserver.
            server.start();
            server.join();
        }
        catch (Exception ex)
        {
            ex.printStackTrace();
        }
    }
 }

and also using the YAML file

global:
  scrape_interval:     2s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 2s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'telemetryserver'

    metrics_path : '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:2018']

I can monitor the metric "metric_1" by running prometheus and then running the query "metric_1". This is all ok, but if I add another metric called metric_2 by uncommenting the lines in the instrumented class then metric_2 shows up on prometheus but metric_1 stops showing up. Further more, if I uncomment the DefaultExports.initialize(); line (which adds all the default prometheus metrics) then both metric_1 and metric_2 both stop showing up and only a default metric called "jvm_threads_current" shows up.

Can someone please enlighten me on how I can monitor multiple metrics in prometheus? Any help appreciated.

Answer 1

Your application is fine.

Just make sure to test it by hitting the /metrics endpoint. This is how prometheus collects the metrics from your application.

When I test it locally, with both metrics and exporting the JVM metrics, I have an output similar to this one:

*** lots of jvm metrics
jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 0.0
jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 0.0
# HELP metric_2 metric_2
# TYPE metric_2 gauge
metric_2 12.0
# HELP metric_1 metric_1
# TYPE metric_1 gauge
metric_1 6.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 16.0

Which contains your custom metrics and also the metrics from JVM.

Your Prometheus configuration is also ok. To test the integration, go to "Status">"Targets", and you should see your service there like in the image bellow:

Then go to "Graph" page and enter the expression {job="telemetryserver"} to check that Prometheus is in fact collecting the metrics from your application. You should see something similar to: