CODEWITHSUNDEEP
javaprometheusprometheus-java
jadler2660
jadler2660

Reputation: 11

Prometheus Counter Inconsistency

I'm using the Prometheus Java simpleclient within a web service to keep track of how many events result in one status or another.

I'm able to check within the logs that the counter is being invoked and is incrementing internally, but it seems that a lot of times the data is not making it to the /metrics endpoint.

For example, just now, after incrementing the counter 3 times for the same status a few minutes apart each, the log would print out "Current Value = 0, New value = 1" three times. The first two times did not show any data on the /metrics endpoint, and after the 3rd increment, it finally showed a value of 1, which means I lost the record of the first 2 events.

The code I have is the following below, besides some name changes.

private static final Counter myCounter = Counter.build()
        .name("myMetric")
        .help("My metric")
        .labelNames("status").register();
...

private static void incrementCounter(String status) {
    Counter.Child counter = myCounter.labels(status);
    Logger.info("Before Incrementing counter for status= " + status + ". Current value=" + counter.get());
    counter.inc();
    Logger.info("After Incrementing counter for status= " + status + ". New value=" + counter.get());
}

I'm at a loss as for why Prometheus doesn't seem to be able to keep track of these counters consistently. Is anyone able to see what's wrong or a better way to record these Counter metrics?

Upvotes: 1

Views: 3231

Answers (1)

Nolequen
Nolequen

Reputation: 4245

The only reason I can guess are concurrent incrementCounter calls. The io.prometheus.client.SimpleCollector#labels method is not thread-safe (despite that children field has ConcurrentMap type), thus it is possible to get different io.prometheus.client.Counter.Child on every call.

As for getting metrics via http - every call to /metrics endpoint leads to io.prometheus.client.Counter#collect method call, which retrieves value of the only one child.

I would suggest you to use your own concurrent map to store counters:

private static final ConcurrentMap<String, Counter.Child> counters = new ConcurrentHashMap<>();   

// ...

private static void incrementCounter(String status) {
  Counter.Child counter = counters.computeIfAbsent(status, myCounter::labels) ;
  // ...
}

Upvotes: 3

Related Questions